What is the ELK Stack?
ELK is an acronym from the first letter of three open-source products — Elasticsearch, Logstash, and Kibana— from Elastic
. The 3 products are used collectively (though can be used separately) mainly for centralizing and visualizing logs from multiple servers (as much as you want).
- Elasticsearch is basically a distributed, NoSQL data store, that uses on the Lucene search capabilities.
- Logstash is a log collection pipeline tool that accepts inputs from various sources (log forwarder), executes different filtering and formatting, and writes the data to Elasticsearch.
- Kibana is a graphical-user-interface (GUI) for visualization of Elasticsearch data.
The ELK Stack is the most widely used log analytics solution, beating Splunk’s enterprise software, which had long been the market leader. The ELK Stack is downloaded 500,000 times every month, making it the world’s most popular log management platform. In contrast, Splunk — the historical leader in the space — self-reports 10,000 total customers.
This tutorial is a guide to set up ELK stack and Filebeat as log-forwarder to gather syslogs of a remote machine (or as many servers as you want).
Continue reading “Install ELK stack on CentOS 7 to centralize logs analytics”
Elasticsearch is a distributed storage and real-time search engine.
- Distributed storage – you just need to setup and add Elasticsearch nodes, it’ll keep the data distributed on the cluster nodes. The distributed-ness makes data durable and highly-available too.
- Real-time search engine – You can get to query the data the moment it’s been written.
Due to the above 2 attributes you have been listening and reading about Elasticsearch, wherever there’s a discussion of real-time data analysis. It’d not be an overstatement to say technologies like Elasticsearch set the foundation for any efficient and reliable search engine.
Continue reading “Install Elasticsearch 5 on CentOS 7.x”
If you have a fresh installation, it is recommended to run the update first
Usually Java comes installed on CentOS 7 (Everything), for CentOS 7 minimal you may need to install Java for various setups. On a CentOS 7 Everything, you can verify it by simply checking the version:
# java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)
Java 1.8.0_111-b15 is the latest available, ‘1.8.0’ states it is Java 8, ‘u111’ means update 111, and ‘b15’means build 15.
For CentOS 7 minimal, the same command will give a different output:
# java -version
-bash: java: command not found
Which means Java is not installed. The latest Java version currently is Java 8.
Continue reading “Install Java 8 on CentOS/RHEL 7.x”
Puppet is an open-source configuration management
tool – for infrastructure orchestration, or automated provisioning, or configuration automation, and lot more. The simplest use case is for automated provisioning i.e. the tasks we need to perform once our machine/VM comes up for the first time (or even after it), like installing webserver, DB server, etc. Instead of manually performing the tasks/running the tasks we can use any of the available configuration management tools (like Puppet) to automate the boring repetitive tasks, and also making configuration consistent across all the servers.
Continue reading “Installing open-source standalone Puppet on CentOS 7”
Vagrant is virtual machine configuration and provisioning utility – uses VirtualBox, VMWare, KVM, etc for virualization and Puppet, Chef, Ansible, or shell scripts for provisioning.
Step 0 – Pre-requisites for Vagrant
The setup just requires VirtualBox to be installed – as Vagrant runs on top of a virtualization software. Here is a previous post – “Install VirtualBox 5.1 for CentOS 7/RHEL 7
“. I am also listing the steps/commands here:
# cd /etc/yum.repos.d/
# wget http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo
# yum update -y
# yum -y install epel-release
# yum -y install gcc make patch dkms qt libgomp
# yum -y install kernel-headers kernel-devel fontforge binutils glibc-headers glibc-devel
# yum -y install VirtualBox-5.1
Step 1 – Install Vagrant
You can download the Vagrant as per your platform
– for this post we’ll be downloading it for CentOS 7. The latest available version is 1.8.6
# wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.rpm
Continue reading “Installing Vagrant on CentOS 7”
(aka Oracle VM VirtualBox
) is an open-source “full virtualization
” solution by Oracle Corporation. Basically it is a hypervisor, which makes it possible to install an unmodified OS on top of your existing host OS – these environments are generally termed as virtual machines
or guest machines
(the base machine the Host). Hypervisors i.e. VirtualBox
provide a virtualization layer i.e. making it possible for the guest machines to access the abstracted hardware resources.
Continue reading “Install VirtualBox 5.1 for CentOS 7/RHEL 7”