DEVOPS / SYS ADMIN Q & A #9

: LINUX SYSTEM / APPLICATION

MONITORING, PERFORMANCE
TUNING, PROFILING METHODS K Hong
google.com/+KHongSanF

& TOOLS Francisc

Follow

3,890 followers

Ph.D. / Golden Gate Ave, San

Francisco / Seoul National Univ /
Carnegie Mellon / UC Berkeley /
DevOps / Deep Learning /
Visualization

(http://www.addthis.com/bookmark.php?
v=250&username=khhong7)

Sponsor Open Source

development activities and free
contents for everyone.

LENOVO YOGA 510-14IKB 14-INCH TOUCHSCREEN

Thank you.

bogotobogo.com site search:

Custom Search Search - K Hong
(http://bogotobogo.com/about_us.php)

Measure twice, cut once!

Note
This article is based on:

1. Linux Performance Tools (https://www.youtube.com/watch?

v=FJW8nGV4jxY)
2. USE Method: Linux Performance Checklist LENOVO IDEAPAD 110
80UD014CIH 15.6-INCH
(http://www.brendangregg.com/USEmethod/use-linux.html) LAPTOP , BLACK TEXTURE

DevOps

Phases of
Continuous
Integration
(/DevOps/Continuous_Integration_Phases.php)

Software
development
methodology
 (/DesignPatterns/software_development_methodology.php)

Introduction to
DevOps
(/DevOps/DevOps_Jenkins_Chef_Puppet_Graphite_Logstash.php)

Samples of
Lenovo ideapad110 80UD00RYIH Continuous
(171) Integration (CI) /
Continuous
Delivery (CD) - Use
cases
(/DevOps/DevOps_CI_CD_Pipeline_Sample.php)

Artifact repository
and repository
Basic tools management
(/DevOps/DevOps_Artifacts_Artifactory_Sonatype_Nexus_Maven_Artifact_Reposi
1. uptime
2. top Linux - General,
3. ps shell
4. vmstat - virtual memory statistics programming,
5. iostat - block I/O disk utilization processes &
6. mpstat - multi-processor statistics signals ...
7. free (/Linux/linux_tips1.php)
8. sar - system activity report
RabbitMQ...
9. strace
(/python/RabbitMQ_Celery/python_Installing_RabbitMQ_Celery.php)
10. dmesg
MariaDB
(/DevOps/DevOps_MariaDB.php)

New Relic APM

with NodeJS :
simple agent setup
on AWS instance
1. uptime - load average (/DevOps/DevOps_NewRelic-
We can get the load average from commands like top or APM-Application-
uptime. Performance-
Management-
If load > # of CPUs, it may mean CPU saturation. setup.php)

Nagios on CentOS
k@laptop:~$ uptime 7 with Nagios
16:48:25 up 32 min, Remote Plugin
2 users, load average: 0.58, 1.13, 2.46
Executor (NRPE)
(/DevOps/DevOps_CentOS_Nagios_Infrastructure_Monitoring_Nagios-
Remote-Plugin-
From left to right, these numbers show us the average load over
Executor-
the last 1 minute, the last 5 minutes, and the last 15 minutes. In NRPE.php)
other words, the above output indicates:
Nagios - The
industry standard
load average over the last 1 minute: 0.58
in IT infrastructure
load average over the last 5 minutes: 1.13
load average over the last 15 minutes: 2.46
monitoring on
Ubuntu
(/DevOps/DevOps_Nagios_Infrastructure_Monitoring_Nagios-
Assuming 1 cpu machine, it means: Remote-Plugin-
Executor-
NRPE.php)
load average over the last 1 minute: 0.58 => The CPU idled for 42% of the time
Zabbix
load average over the last 5 minutes: 1.13 => .13 processes were 3 install
waiting foronthe CPU
load average over the last 15 minutes: 2.46 => On average, 1.46 processes were waiting for the CPU
Ubuntu 14.04 &
adding hosts /
items / graphs
Actually, if the machine has 2 CPUs, then it would mean: (/DevOps/DevOps-
Zabbix3-Server-
and-Agent-Install-
Ubuntu14-Adding-
load average over the last 1 minute: 0.58 => The CPU idled for 142% of the time
Hosts-Items-
load average over the last 5 minutes: 1.13 => .87 processes were waiting for the CPU
load average over the last 15 minutes: 2.46 => On average, 0.46 processes were waiting for the CPU
Graphs.php)

Datadog -
Monitoring with
PagerDuty/HipChat
and APM
(/DevOps/DevOps-

2. top - system and per-process

2. top - system and per-process Monitoring-with-
Datadog-
interval summary PagerDuty-
When we use top to diagnose load, the basic steps are to HipChat.php)
examine the top output to identify what resources we are
Install and
running out of (CPU, RAM, disk I/O). Once we have gured that
Con gure Mesos
out, we can try to identify what processes are consuming those Cluster
resources the most. (/DevOps/DevOps_Mesos_Install.php)

%CPU is summed across all CPUs Cassandra on a

Single-Node
Cluster
Can miss short-lived processes (atop won't)
(/DevOps/DevOps-
Cassandra-On-A-
This section is a compiled work from the following sources: Single-Node-
Cluster.php)
1. Top (http://man7.org/linux/man-pages/man1/top.1.html)
2. Understanding Linux CPU stats Container
(http://blog.scoutapp.com/articles/2015/02/24/understanding- Orchestration :
Docker Swarm vs
linuxs-cpu-stats)
Kubernetes vs
Apache Mesos
(/DevOps/DevOps-
The top program provides a dynamic real-time view of a running Docker-Swarm-vs-
system. It can display system summary information, as well as a Kubernetes-vs-
list of processes or threads currently being managed by the Apache-
kernel. Mesos.php)

Descriptions for the top display:
%Cpu(s): 10.7 us, 2.9 sy,
OpenStack install
on Ubuntu 16.04
server - DevStack
0.0 ni, 85.7 id, 0.5 wa, (/DevOps/OpenStack-
0.0 hi, 0.2 si, 0.0 st
Install-On-Ubuntu-
16-Server.php)

This line shows CPU state percentages based on the interval AWS EC2
since the last refresh. Container Service
(ECS) & EC2
1. us, user user cpu time (or) % CPU time spent in user space, Container Registry
time running un-niced user processes. (ECR) | Docker
Registry
Shells, compilers, databases, web servers, and the
(/DevOps/DevOps-
programs associated with the desktop are all user space
ECS-ECR.php)
processes. If the processor isn't idle, it is quite normal that
the majority of the CPU time should be spent running user CI/CD with CircleCI
space processes. - Heroku deploy
2. sy, system system cpu time (or) % CPU time spent in kernel (/DevOps/DevOps-
space. This is the amount of time that the CPU spent CircleCI-Heroku-
running the kernel. All the processes and system resources Deploy.php)
are handled by the Linux kernel. When a user space process
Introduction to
needs something from the system, for example when it
Terraform with
needs to allocate memory, perform some I/O, or it needs to
AWS elb & nginx
create a child process, then the kernel is running. In fact the (/DevOps/DevOps-
scheduler itself which determines which process runs next Terraform.php)
is part of the kernel. The amount of time spent in the kernel
should be as low as possible. In this case, just 2.9% of the Kubernetes I -
time given to the di erent processes was spent in the Running
kernel. This number can peak much higher, especially when Kubernetes Locally
via Minikube
there is a lot of I/O happening.
(/DevOps/DevOps-
3. ni, nice time running niced user processes.
Kubernetes-1-
Niceness is a way to tweak the priority level of a process so Running-
that it runs less frequently. The niceness level ranges from Kubernetes-
-20 (most favorable scheduling) to 19 (least favorable). By Locally-via-
default processes on Linux are started with a niceness of 0. Minikube.php)
A "niced" process is one with a positive nice value. So if the
processor's nice value is high, that means it is working with Kubernetes II -
kops on AWS
some low priority processes. So this indicator is useful when
(/DevOps/DevOps-
we see high CPU utilization and we are afraid that this high
Kubernetes-II-
load will have bad e ect on our system: kops-on-AWS.php)
1. High CPU utilization with high nice value: Nothing to
worry, not so important tasks doing there job, Kubernetes III -
important processes will easily get CPU time if they kubeadm on AWS
need. This situation is not a real bottleneck. (/DevOps/DevOps-
 2. High CPU utilization with low nice value: Something to Kubernetes-III-
worry because the CPU is stressed with important Kubernetes-on-
processes so these or new processes will have to wait. Linux-with-
kubeadm.php)
This situation is a real bottleneck.
4. id, idle time spent in the kernel idle handler.
The id statistic tell us that the processor was idle just over
85.7% of the time during the last sampling period. The total
DEVOPS /
of the user space percentage - us, the niced percentage - ni,
and the idle percentage - id, should be close to 100%. Which
SYS ADMIN Q
it is in this case. If the CPU is spending a more time in the &A
other states then something is probably wrong, and may
need trouble shooting.
5. wa, IO-wait time waiting for I/O completion.
(1A) - Linux
I/O operations are slow compared to the speed of a CPU. Commands
There are times when the processor has initiated a read or (/DevOps/DevOps-
write operation and then it has to wait for the result, but Sys-Admin-
has nothing else to do. In other words it is idle while waiting Interview-
for an I/O operation to complete. The time the CPU spends Questions-
in this state is shown by the 'wa' statistic. Commands.php)
'wa' is the measure of time over a given period that a CPU
(1B) - Linux
spent idle because all runnable tasks were waiting for a IO
Commands
operation to be ful lled.
(/DevOps/DevOps-
6. hi time spent servicing hardware interrupts. Sys-Admin-
This is the time spent processing hardware interrupts. Interview-
Hardware interrupts are generated by hardware devices Questions-
(network cards, keyboard controller, external timer, Commands-2.php)
hardware senors, etc.) when they need to signal something
(2) - Networks
to the CPU (data has arrived for example). Since these can
(/DevOps/DevOps-
happen very frequently, and since they essentially block the
Sys-Admin-
current CPU while they are running, kernel hardware
Interview-
interrupt handlers are written to be as fast and simple as Questions-
possible. Networks.php)
On a system where no processes have been niced then the
number will be 0. (2B) - Networks
Hardware interrupts are physical interrupts sent to the CPU (/DevOps/DevOps-
from various peripherals like disks and network interfaces. Sys-Admin-
Interview-
Software interrupts come from processes running on the
Questions-
system. A hardware interrupt will actually cause the CPU to
Networks-2.php)
stop what it is doing and go handle the interrupt. A software
interrupt doesn't occur at the CPU level, but rather at the (3) - Linux Systems
kernel level. (/DevOps/DevOps-
7. si time spent servicing software interrupts. Sys-Admin-
This represents the time spent in softirqs Interview-
Questions-
(http://zeniv.linux.org.uk/pub/linux/people/willy/willy/lca/paper.pdf).
Systems.php)
8. st time stolen from this vm by the hypervisor.
This represents "steal time", and it is only relevant in
(4) - Scripting
virtualized environments. It represents time when the real
(Ruby/Shell)
CPU was not available to the current virtual machine - it was (/DevOps/DevOps-
"stolen" from that VM by the hypervisor (either to run Sys-Admin-
another VM, or for its own needs). Interview-
This number tells how long the virtual CPU has spent Questions-
waiting for the hypervisor to service another virtual CPU Scripting.php)
running on a di erent virtual machine. Since in the real-
(5) - Con guration
world these virtual processors are sharing the same
Management
physical processor(s) then there will be times when the
(/DevOps/DevOps-
virtual machine wanted to run but the hypervisor scheduled Sys-Admin-
another virtual machine instead. Interview-
Questions-
Here are some of the trouble shootings: Con gurations.php)

(6) - AWS VPC

1. High user mode CPU usage - If a system suddenly jumps
setup
from having spare CPU cycles to running at out high, then
(public/private
the rst thing to check is the amount of time the CPU subnets with NAT)
spends running user space processes. If this is high, then it (/DevOps/DevOps-
probably means that a process has gone crazy and is eating Sys-Admin-
up all the CPU time. Interview-
Using the top command we will be able to see which Questions-AWS-
process is to blame and restart the service or kill the VPC-Setup.php)

process.
 2. High kernel CPU usage - Sometimes this is acceptable. For (6B) - AWS VPC
example, a program that does lots of console I/O can cause Peering
the kernel usage to spike. However if it remains higher for (/DevOps/DevOps-
Sys-Admin-
long periods of time, then it could be an indication that
Interview-
something isn't right.
Questions-AWS-
A possible cause of such spikes could be a problem with a VPC-Peering.php)
driver/kernel module.
3. High niced value CPU usage - If the amount of time the (7) - Web server
CPU is spending running processes with a niced priority (/DevOps/DevOps-
value jumps, then it means that someone has started some Sys-Admin-
intensive CPU jobs on the system, but they have niced the Interview-
Questions-Web-
task.
HTTP.php)
If the niceness level is greater than zero, then the user has
been courteous enough lower to the priority of the process
(8) - Database
and therefore avoid a CPU overload. There is probably little (/DevOps/DevOps-
that needs to be done in this case, other than maybe nd Sys-Admin-
out who has started the process. Interview-
But if the niceness level is less than 0, then we will need to Questions-
investigate what is happening and who is responsible, as Database.php)
such a task could easily cripple the responsiveness of the
(9) - Linux System /
system.
Application
4. High waiting on I/O This means that there are some
Monitoring,
intensive I/O tasks running on the system that don't use up Performance
much CPU time. If this number is high for anything other Tuning, Pro ling
than short bursts, then it means that either the I/O Methods & Tools
performed by the task is very ine cient, or the data is being (/DevOps/DevOps-
transferred to a very slow device, or there is a potential Sys-Admin-
problem with a hard disk that is taking a long time to Interview-
Questions-Linux-
process reads & writes.
Monitoring-
5. High interrupt processing This could be an indication of a
System-
broken peripheral that is causing lots of hardware Application-
interrupts or of a process that is issuing lots of software Performance-
interrupts. Tuning-Tools.php)
6. Large stolen time Basically, this means that the host
system running the hypervisor is too busy. If possible, check (10) - Trouble
the other virtual machines running on the hypervisor, Shooting: Load,
Throughput,
and/or migrate our virtual machine to another host.
Response time and
Leaks
When we have a slow server, one of the rst values we should (/DevOps/DevOps-
look at is I/O wait so we can rule out disk I/O. If I/O wait is low, Sys-Admin-
then we can look at the idle percentage. If I/O wait is high, then Interview-
the next step is to diagnose what is causing high disk I/O. Questions-
Trouble-Shooting-
Slow-Application-
If I/O wait and idle times are low, then we will likely see a high
Performance-
user time percentage, so we must diagnose what is causing high
BottleNecks-
user time. If the I/O wait is low and the idle percentage is high, Leaks.php)
we then know any sluggishness is not because of CPU resources,
and we will have to start troubleshooting elsewhere. (11) - SSH key
pairs, SSL
This might mean looking for network problems, or in the case of Certi cate, and SSL
a web server, looking at slow queries to MySQL, for instance. Handshake
(/DevOps/DevOps-
Sys-Admin-
Interview-
Questions-SSH-
Connection-SSL-
Certi cates.php)

3. ps - process status listing (12) - Why is the

database slow?
$ ps -ef f (/DevOps/DevOps-
... Sys-Admin-
root 979 1 0 Nov12 ? Ss 0:03 Interview-
/lib/systemd/systemd-logind
avahi 990 1 0 Nov12 ? S 0:03 avahi-daemon: running [laptop.local]
Questions-Why-is-
avahi 992 990 0 Nov12 ? S 0:00 \_ avahi-daemon: chroot helper
database-
root 1035 1 0 Nov12 ? Ssl 0:42 NetworkManager
nobody 1556 1035 0 Nov12 ? S 0:21 slow.php)
\_ /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfa
root 30549 1035 0 Nov13 ? S 0:00 \_ /sbin/dhclient -d -sf /usr/lib/NetworkManager/nm-dhcp-client.action -pf /run
root 1049 1 0 Nov12 ? Sl (13) - Is my web
0:03 /usr/lib/policykit-1/polkitd --no-debug
root 1051 1 0 Nov12 tty4 Ss+ site-8down?
0:00 /sbin/getty 38400 tty4
...
(/DevOps/DevOps-
Sys-Admin-
Note that we added "f" which displays ASCII art process Interview-
hierarchy (forest). Questions-Is-
Website-
down.php)
To get the top 5 cpu eating process:

(14) - Is my server
down?
$ ps -eo pcpu,pid,user | sort -k1 -r | head -6
%CPU PID USER COMMAND (/DevOps/DevOps-
5.8 3444 k /opt/google/chrome/chrome Sys-Admin-
3.1 27831 k /opt/google/chrome/chrome --type=rendererInterview-
...
2.5 982 root Questions-Is-
/usr/lib/xorg/Xorg -core :0 -seat seat0 ...
2.5 3544 k /opt/google/chrome/chrome --type=gpu-process ...
Server-down.php)
22.9 3645 k /opt/google/chrome/chrome --type=renderer -- ...

(15) - Why is the

server sluggish?
Note that we used 'r' for 'sort' to reverse, and '6' for 'head' to
(/DevOps/DevOps-
include the column labels.
Sys-Admin-
Interview-
Questions-Why-is-
theServer-
slow.php)

(16A) - Serving
multiple domains
4. vmstat using Virtual Hosts
The vmstat tool provides information about memory, swap - Apache
utilization, IO wait, and system activity. It is particularly useful for (/DevOps/DevOps-
diagnosing I/O-related issues. Sys-Admin-
Interview-
Questions-Serving-
Usage : vmstat [interval [count]]
Multiple-Domains-
Using-Virtual-
Hosts-Apache.php)
vmstat 1 20

(16B) - Serving
This runs a vmstat every second(1), twenty times(20). This gives multiple domains
a pretty good sample of the current state of the system. The using server block
- Nginx
output generated should look like the following:
(/DevOps/DevOps-
Sys-Admin-
$ vmstat 1 5 Interview-
Questions-Serving-
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in Multiple-Domains-
cs us sy id wa st
0 0 1346096 220912 19284 303804 21 18 95 65 61Using-Virtual-
142 35 8 55 1 0
1 0 1346096 220940 19284 303804 0 0 0 0 785 1612 15 3 82 0 0
Hosts-Nginx.php)
1 0 1346092 220908 19292 303808 0 0 0 56 876 1715 11 4 84 2 0
2 0 1346092 220908 19292 303808 0 0 0 0 678 1355 10 3 88 0 0
1 0 1346092 220940 19292 303808 0 0 0 0 761(16C)
1568- Reverse
14 5 81 0 0
proxy servers and
load balancers -
Nginx
The rst output line has some summary since boot values.
(/DevOps/DevOps-
Sys-Admin-
The rst column, r is runnable tasks. Interview-
Questions-
The memory and swap columns provide the same kind of Reverse-proxy-
information provided by the free -m command, though in a servers-and-load-
balancing-
slightly more di cult to comprehend format. The most salient
Nginx.php)
information produced by this command is the wa column, which
is the nal column in most implementations. This eld displays (17) - Linux startup
the amount of time the CPU spends waiting for IO process
operations to complete. (/DevOps/DevOps-
Sys-Admin-
If this number is consistently and considerably higher than 0, we Interview-
Questions-Linux-
might consider taking measures to address the IO usage.
Boot-Startup-
Process.php)

1. Procs: (18) - phpMyAdmin

r: The number of processes waiting for run time. with Nginx virtual
b: The number of processes in uninterruptible sleep. host as a
2. Memory subdomain
swpd: the amount of virtual memory used. (/DevOps/DevOps_phpMyAdmin_Nginx_Subdomain.php)
free: the amount of idle memory.
(19) - How to SSH
bu : the amount of memory used as bu ers.
login without
cache: the amount of memory used as cache.
password?
 inact: the amount of inactive memory. (-a option) (/DevOps/DevOps-
active: the amount of active memory. (-a option) Sys-Admin-
3. Swap Interview-
Questions-SSH-
si: Amount of memory swapped in from disk (/s).
login-without-
so: Amount of memory swapped to disk (/s).
password.php)
4. IO
bi: Blocks received from a block device (blocks/s). (20) - Log Rotation
bo: Blocks sent to a block device (blocks/s). (/DevOps/DevOps-
5. System Sys-Admin-
in: The number of interrupts per second, including the Interview-
clock. Questions-Log-
Rotation.php)
cs: The number of context switches per second.
6. CPU
(21) - Monitoring
These are percentages of total CPU time. Metrics
us: Time spent running non-kernel code. (user time, (/DevOps/DevOps-
including nice time). Sys-Admin-
sy: Time spent running kernel code. (system time). Interview-
id: Time spent idle. Prior to Linux 2.5.41, this includes IO- Questions-
wait time. Monitoring-
wa: Time spent waiting for IO. Prior to Linux 2.5.41, Metrics.php)

included in idle.
(22) - lsof
st: Time stolen from a virtual machine. Prior to Linux 2.6.11,
(/DevOps/DevOps-
unknown. Sys-Admin-
Interview-
Questions-
For more on vmstat, please visit Linux Performance lsof.php)
Measurements using vmstat (https://www.thomas-
(23) - Wireshark
krenn.com/en/wiki/Linux_Performance_Measurements_using_vmstat)
introduction
(/DevOps/DevOps-
WireShark-
Tutorial-
Introduction.php)

(24) - User account

5. iostat - Block I/O disk utilization management
(/DevOps/DevOps-
To monitor disk read/write rates of individual disks, we can use
Sys-Admin-
iostat. This tool allows us to monitor I/O statistics for each Interview-
device or partition. Using iostat command, we can nd out disk Questions-Linux-
utilization and monitor system input/output device loading by User-Account-
observing the time the physical disks are active in relation to Management.php)
their average transfer rates.
(25) - Domain
Name System
To use this tool, we need to run sysstat package.
(DNS)
(/DevOps/DevOps-
To install sysstat on Ubuntu or Debian: Sys-Admin-
Interview-
Questions-
$ sudo apt-get install sysstat DNS.php)

(26) - NGINX
Syntax for disk utilization report looks like this:
SSL/TLS, Caching,
and Session
(/DevOps/DevOps-
iostat -d -x interval count
Sys-Admin-
Interview-
where: Questions-NGINX-
SSL-TLS-Caching-
Session.php)
1. -d : Display the device utilization report (d == disk)
2. -x : Display extended statistics including disk utilization
(0) - Linux Sys
3. interval : It is time period in seconds between two samples. Admin's Day to
iostat 2 will give data at each 2 seconds interval. Day tasks
4. count : It is the number of times the data is needed. iostat 2 (/DevOps/DevOps-
5 will give data at 2 seconds interval 5 times. Sys-Admin-
Interview-
Questions-Day-To-
Day-Tasks.php)
 $ iostat -d -x 5 3
Linux 3.13.0-40-generic (laptop) 10/14/2015 _x86_64_Sponsor Open Source
(2 CPU)
development activities and free
contents for everyone.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.75 4.78 6.15 2.13 104.99 45.86 36.45 0.27 32.58 22.74 61.06 3.03 2.51

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz

Thank you. avgqu-sz await r_await w_await svctm %util
sda 0.00 5.20 0.00 7.80 0.00 80.00 20.51 0.14 17.74 0.00 17.74 12.41 9.68
- K Hong
Device: rrqm/s wrqm/s r/s w/s rkB/s (http://bogotobogo.com/about_us.php)
wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 4.20 4.40 0.80 2.80 20.00 47.20 37.33 0.11 31.11 76.00 18.29 31.11 11.20

The following values from the iostat output are the major ones:

1. r/s : The number of read requests per second. See if a hard

disk reports consistently high reads
2. w/s : The number of write requests per second. See if a
hard disk reports consistently high writes
3. svctm : The average service time (in milliseconds) for I/O
requests that were issued to the device.
4. %util : Percentage of CPU time during which I/O requests
were issued to the device (bandwidth utilization for the
device). Device saturation occurs when this value is close to
100%.

LENOVO YOGA 510-14IKB

14-INCH TOUCHSCREEN
LAPTOP , BLACK

6. mpstat - multi-processor
statistics, per-CPU
We may want to look for unbalanced workloads, hot CPUs:

$ mpstat -P ALL 1

Jenkins
Linux 3.13.0-40-generic (laptop) 11/14/2015 _x86_64_ (2 CPU)

12:28:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
12:28:06 AM all 16.49 0.00 6.19 0.00 0.00 0.00 0.00 0.00 0.00 77.32
12:28:06 AM 0 16.67 0.00 6.25 0.00 0.00 0.00 0.00 0.00 0.00 77.08
12:28:06 AM 1 16.16 0.00 7.07 0.00 0.00 0.00 0.00 0.00 0.00 76.77
Install
12:28:06 AM CPU %usr %nice %sys %iowait %irq
(/DevOps/Jenkins/Jenkins_Install.php)
%soft %steal %guest %gnice %idle
12:28:07 AM all 22.96 0.00 7.65 1.02 0.00 0.00 0.00 0.00 0.00 68.37
12:28:07 AM 0 21.65 0.00 7.22 2.06 0.00 Con guration
0.00 0.00 - 0.00 0.00 69.07
12:28:07 AM 1 25.00 0.00 8.00 1.00 0.00 Manage0.00
0.00 Jenkins -0.00 0.00 66.00
security setup
(/DevOps/Jenkins/Jenkins_Con guration.php)

Adding job and

build
(/DevOps/Jenkins/Jenkins_Adding_Job_and_Build.php)

Scheduling jobs
7. free (/DevOps/Jenkins/Jenkins_Scheduling_Job.php)

$ free -m Managing_plugins
total used free shared buffers cached
(/DevOps/Jenkins/Jenkins_Manage_Plugins.php)
Mem: 3545 3424 120 118 1 234
-/+ buffers/cache: 3188 357
Swap: 3681 1309 2372 Git/GitHub plugins,
SSH keys
con guration, and
Fork/Clone
The m option displays all data in MBs. The total 3545 MB is the (/DevOps/Jenkins/Jenkins_Git_Github.php)
total amount of RAM installed on the system, that is 3.5GB. The
used column shows the amount of RAM that has been used by JDK & Maven setup
linux, in this case around 3.4 GB. The output is pretty self (/DevOps/Jenkins/Jenkins_Maven_Setup.php)
explanatory. Notable columns are the cached and bu ers
columns. The second line tells that 0.3 GB is free. This is the free Build con guration
memory in rst line added with the bu ers and cached amount for GitHub Java
application with
of memory.
Maven
(/DevOps/Jenkins/Jenkins_GitHub_Java_Application_Project_Build_Con guration_
The last line is the swap memory, which in this case is 2.3GB Build Action for
free. GitHub Java
application with
Maven - Console
Output, Updating
Maven
(/DevOps/Jenkins/Jenkins_GitHub_Java_Application_Project_Build_Action.php)

Commit to
8. sar - system activity report changes to GitHub
& new test results -
sar -P ALL 1 2 displays real time CPU usage for ALL cores every 1
Build Failure
second for 2 times (broken down by all cores). (/DevOps/Jenkins/Jenkins_GitHub_Java_Application_Commit_Changes.php)

Commit to
$ sar -P ALL 1 2
Linux 3.13.0-40-generic (laptop) 11/14/2015
changes to GitHub
_x86_64_ (2 CPU)
& new test results -
08:34:18 AM CPU %user %nice %system %iowait Successful
%steal Build
%idle
08:34:19 AM all 2.58 0.00 1.03 0.00 (/DevOps/Jenkins/Jenkins_GitHub_Java_Application_Fixing_Build_Failures.php)
0.00 96.39
08:34:19 AM 0 3.09 0.00 1.03 0.00 0.00 95.88
08:34:19 AM 1 2.04 0.00 1.02 0.00 0.00 96.94
Adding code
08:34:19 AM CPU %user %nice %system %iowait coverage
%steal and %idle
08:34:20 AM all 4.57 0.00 2.54 0.00 metrics
0.00 92.89
08:34:20 AM 0 4.08 0.00 2.04 0.00 (/DevOps/Jenkins/Jenkins_Adding_Code_Coverage_and_Metrics.php)
0.00 93.88
08:34:20 AM 1 5.10 0.00 3.06 0.00 0.00 91.84

Jenkins on EC2 -
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 3.58 0.00 1.79 0.00 creating
0.00an EC294.63
Average: 0 3.59 0.00 1.54 0.00 account,
0.00ssh to94.87
Average: 1 3.57 0.00 2.04 0.00 EC2, 0.00
and install94.39
Apache server
(/DevOps/Jenkins/Jenkins_on_EC2_1_setting_up_instance.php)
sar -P 1 1 2 displays real time CPU usage for core number 1,
Jenkins on EC2 -
every 1 second for 2 times.
setting up Jenkins
account, plugins,
and Con gure
$ sar -P 1 1 2
Linux 3.13.0-40-generic (laptop) 11/14/2015 System
_x86_64_ (2 CPU)
(JAVA_HOME,
08:35:45 AM CPU %user %nice %system %iowait MAVEN_HOME,
%steal %idle
08:35:46 AM 1 2.06 0.00 1.03 0.00 noti 0.00
cation email)
96.91
08:35:47 AM 1 5.10 0.00 3.06 0.00 0.00 91.84
(/DevOps/Jenkins/Jenkins_on_EC2_2_Jenkins_Con guration.php)
Average: 1 3.59 0.00 2.05 0.00 0.00 94.36

Jenkins on EC2 -
Creating a Maven
sar -n DEV 1 1 displays network devices vital statistics for eth0, project
eth1, etc. every 1 second for 1 times. (/DevOps/Jenkins/Jenkins_on_EC2_3_Jenkins_Creating_Maven_Project.php)

Jenkins on EC2 -
$ sar -n DEV 1 1 Con guring
Linux 3.13.0-40-generic (laptop) 11/14/2015 _x86_64_
GitHub Hook(2andCPU)

Noti cation service

08:38:43 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
08:38:44 AM vmnet8 0.00 0.00 0.00 0.00to Jenkins
0.00server 0.00 0.00 0.00
08:38:44 AM eth0 0.00 0.00 0.00 0.00for any0.00
changes to0.00 0.00 0.00
08:38:44 AM lo 0.00 0.00 0.00 0.00the repository
0.00 0.00 0.00 0.00
08:38:44 AM lxcbr0 0.00 0.00 0.00 0.00(/DevOps/Jenkins/Jenkins_on_EC2_4_GitHubHook_Noti
0.00 0.00 0.00 0.00 cation_to_Jenkins_Server
08:38:44 AM wlan0 22.00 1.00 3.17 0.20 0.00 0.00 0.00 0.00
08:38:44 AM vmnet1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
08:38:44 AM docker0 0.00 0.00 0.00 0.00Jenkins0.00
on EC2 - 0.00 0.00 0.00
Line Coverage with
Average: IFACE rxpck/s txpck/s rxkB/s txkB/sJaCoCo plugin txcmp/s rxmcst/s
rxcmp/s %ifutil
Average: vmnet8 0.00 0.00 0.00 0.00(/DevOps/Jenkins/Jenkins_on_EC2_5_Line_Coverage_JaCoCo.php)
0.00 0.00 0.00 0.00
Average: eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: lxcbr0 0.00 0.00 0.00 0.00
Setting0.00
up Master0.00 0.00 0.00
Average: wlan0 22.00 1.00 3.17 0.20and Slave nodes 0.00
0.00 0.00 0.00
Average: vmnet1 0.00 0.00 0.00 0.00(/DevOps/Jenkins/Jenkins_on_EC2_setting_up_master_slaves.php)
0.00 0.00 0.00 0.00
Average: docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Jenkins Build
Pipeline &
Dependency
Graph Plugins
(/DevOps/Jenkins/Jenkins_Build_Pipeline_Dependency_Graph_Plugin.php)

Jenkins Build Flow

Plugin
9. strace (/DevOps/Jenkins/Jenkins_Build_Flow_Plugin.php)
The strace is the tool that helps in debugging issues by tracing
Jenkins Setting up
system calls executed by a program.
Slave nodes on
Here are the samples of strace command: AWS
(/DevOps/Jenkins/Jenkins_Slave_Nodes_AWS.php)

# Slow the target command and print details for each syscall:
strace command

# Slow the target PID and print details for each syscall:
strace -p PID

Puppet
# Slow the target PID and any newly created child process, printing syscall details:
strace -fp PID

# Slow the target PID and record syscalls, printing a summary:

strace -cp PID
Puppet with
# Slow the target PID and trace open() syscalls only: Amazon AWS I -
strace -eopen -p PID Puppet accounts
(/DevOps/Puppet/puppet_amazon_aws.php)
# Slow the target PID and trace open() and stat() syscalls only:
strace -eopen,stat -p PID
Puppet with
Amazon
# Slow the target PID and trace connect() and accept() syscalls only:AWS II
strace -econnect,accept -p PID (ssh &
puppetmaster/puppet
# Slow the target command and see what other programs it launches (slow them too!):
install)
strace -qfeexecve command
(/DevOps/Puppet/puppet_amazon_aws_2_ssh_puppetmaster_puppet_install.php
# Slow the target PID and print time-since-epoch with (distorted) microsecond resolution:
strace -ttt -p PID Puppet with
Amazon AWS III -
# Slow the target PID and print syscall durations with (distorted) microsecond resolution:
Puppet running
strace -T -p PID
Hello World
(/DevOps/Puppet/puppet_amazon_aws_3_running_puppet.php)

Puppet Code
The strace command allows us to trace the system calls made Basics -
by a program. This is useful for debugging, or simply to nd out Terminology
what a program is doing. By default, strace writes its output to (/DevOps/Puppet/puppet_basics_of_code_terminology.php)
stderr, but we can change this using the -o lename option -
Puppet with
from The Linux Programming Interface.
Amazon AWS on
CentOS 7 (I) -
$ strace date
Master setup on
execve("/bin/date", ["date"], [/* 118 vars */]) = 0 EC2
brk(0) = 0x18b5000 (/DevOps/Puppet/puppet_amazon_aws_ssh_puppetmaster_install_CENTOS7.php
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f809a459000
Puppet with
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
Amazon AWS on
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=154081, ...}) = 0 CentOS 7 (II) -
... Con guring a
close(1) = 0 Puppet Master
munmap(0x7f809a458000, 4096) = 0 Server with
close(2) = 0
Passenger and
exit_group(0) = ?
+++ exited with 0 +++ Apache
(/DevOps/Puppet/puppet_amazon_aws_ssh_puppetmaster_install_CENTOS7_2_P

Puppet master
Each system call is displayed in the form of a function call, with /agent ubuntu
both input and out- put arguments shown in parentheses. 14.04 install on
EC2 nodes
After the closing parenthesis of the traced call, strace prints an (/DevOps/Puppet/puppet_install_on_ubuntu_14_04_trusty.php)
equal sign ( = ), fol- lowed by the return value of the system call.
If the system call failed, the symbolic errno value is also Puppet master
post install tasks -
displayed. Thus, we see ENOENT displayed for the failure of the
master's names
access() call above.
and certi cates
setup,
Even for a simple program, the output produced by strace is (/DevOps/Puppet/puppet_master_post_install_tasks_on_ubuntu_14_04_trusty.ph
made voluminous by the system calls executed by the C run-
time startup code and the loading of shared libraries. For a Puppet agent post
complex program, the strace output can be extremely long. install tasks -
con gure agent,
hostnames, and
For these reasons, it is sometimes useful to selectively lter the
sign request
output of strace. (/DevOps/Puppet/puppet_agent_post_install_tasks_on_ubuntu_14_04_trusty.php

EC2 Puppet
master/agent basic
tasks - main
manifest with a le
resource/module
 and immediate
$ strace date 2>&1 | grep open
execution on an
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
agent node
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = (/DevOps/Puppet/puppet_basic_tasks_with_ec2_master_agent_nodes.php)
3
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
Setting up puppet
master and agent
with simple scripts
Another method is to use the -e option to select the events to be
on EC2 / remote
traced. For example, we can use the following command to trace
install from
open() and close() system calls: desktop
(/DevOps/Puppet/puppet_setting_up_master_and_agent_with_scripts_EC2.php)

$ strace -e trace=open,close date

EC2 Puppet -
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
close(3) = 0 Install lamp with a
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) =manifest
3 ('puppet
close(3) = 0 apply')
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = (/DevOps/Puppet/puppet_amazon_ec2_Install_LAMP_with_a_Single_Manifest.php
3
close(3) = 0
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
close(3) = 0 EC2 Puppet -
Sun Nov 29 14:40:08 PST 2015 Install lamp with a
close(1) = 0 module
close(2) = 0 (/DevOps/Puppet/puppet_amazon_ec2_Install_LAMP_with_a_Module.php)
+++ exited with 0 +++

Puppet variable
scope
(/DevOps/Puppet/puppet_variable_Scope.php)

Puppet packages,
services, and les
(/DevOps/Puppet/puppet_packages_ les_services.php)

10. dmesg Puppet packages,

The dmesg command displays all messages from the kernel ring services, and les II
bu er which is a data structure that records messages related to with nginx
the operation of the kernel. A ring bu er is a special kind of (/DevOps/Puppet/puppet_packages_ les_services2_nginx.php)
Puppet templates
bu er that is always a constant size, removing the oldest
(/DevOps/Puppet/puppet_templates.php)
messages when new messages come in.
Puppet creating
We can use dmesg command to check why a process was killed. and managing
That happens if the process was consuming too much memory, user accounts with
and the kernel "Out of Memory" (OOM) killer will automatically SSH access
kill the o ending process. (/DevOps/Puppet/puppet_creating_managing_user_accounts_ssh_control_user_p

Puppet Locking
$ dmesg | less user accounts &
[ 54.125380] deploying
Out of memory: Kill process 8320 (stress-ng-brk) sudoers
score 324 or sacrifice child
[ 54.125382] le
Killed process 8320 (stress-ng-brk) total-vm:1309660kB, anon-rss:1287796kB, file-rss:76kB
[ 54.522906] gmain invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
(/DevOps/Puppet/puppet_locking_user_accounts_deploying_sudoers_ le.php)
[ 54.522908] gmain cpuset=accounts-daemon.service mems_allowed=0
...
Puppet exec
resource
(/DevOps/Puppet/puppet_exec_resources.php)

Puppet classes and

modules
(/DevOps/Puppet/puppet_classes_modules.php)

Puppet Forge
modules
(/DevOps/Puppet/Puppet_Forge_Modules.php)
DevOps Puppet Express
Phases of Continuous Integration (/DevOps/Puppet/puppet_express_in_one_day.php)
(/DevOps/Continuous_Integration_Phases.php)
Software development methodology Puppet Express 2
(/DesignPatterns/software_development_methodology.php) (/DevOps/Puppet/puppet_express_in_one_day_2.php)
Introduction to DevOps
(/DevOps/DevOps_Jenkins_Chef_Puppet_Graphite_Logstash.php) Puppet 4 : Changes
Samples of Continuous Integration (CI) / Continuous (/DevOps/Puppet/puppet4_changes.php)
Delivery (CD) - Use cases
(/DevOps/DevOps_CI_CD_Pipeline_Sample.php) Puppet --
Artifact repository and repository management con gprint
(/DevOps/DevOps_Artifacts_Artifactory_Sonatype_Nexus_Maven_Artifact_Repository_Apache_Archiva.php)
(/DevOps/Puppet/puppet_con gprint.php)
Linux - General, shell programming, processes & signals ...
(/Linux/linux_tips1.php) Puppet with
 RabbitMQ... Docker
(/python/RabbitMQ_Celery/python_Installing_RabbitMQ_Celery.php)(/DevOps/Docker/Docker_puppet.php)
MariaDB (/DevOps/DevOps_MariaDB.php)
New Relic APM with NodeJS : simple agent setup on AWS
instance (/DevOps/DevOps_NewRelic-APM-Application-
Performance-Management-setup.php)
Nagios on CentOS 7 with Nagios Remote Plugin Executor
(NRPE)
(/DevOps/DevOps_CentOS_Nagios_Infrastructure_Monitoring_Nagios-
Remote-Plugin-Executor-NRPE.php)
Chef
Nagios - The industry standard in IT infrastructure
monitoring on Ubuntu
(/DevOps/DevOps_Nagios_Infrastructure_Monitoring_Nagios- What is Chef?
Remote-Plugin-Executor-NRPE.php) (/DevOps/Chef/Chef_What_is_Chef.php)
Zabbix 3 install on Ubuntu 14.04 & adding hosts / items /
graphs (/DevOps/DevOps-Zabbix3-Server-and-Agent-Install- Chef install on
Ubuntu14-Adding-Hosts-Items-Graphs.php) Ubuntu 14.04 -
Datadog - Monitoring with PagerDuty/HipChat and APM Local Workstation
(/DevOps/DevOps-Monitoring-with-Datadog-PagerDuty- via omnibus
HipChat.php) installer
Install and Con gure Mesos Cluster (/DevOps/Chef/Install_Chef_on_Ubuntu_14_04_Workstation_Omnibus_Installer.p
(/DevOps/DevOps_Mesos_Install.php)
Cassandra on a Single-Node Cluster (/DevOps/DevOps- Setting up Hosted
Cassandra-On-A-Single-Node-Cluster.php) Chef server
OpenStack install on Ubuntu 16.04 server - DevStack (/DevOps/Chef/Chef_Setting_up_Hosted_Chef_server.php)
(/DevOps/OpenStack-Install-On-Ubuntu-16-Server.php)
AWS EC2 Container Service (ECS) & EC2 Container Registry VirtualBox via
(ECR) | Docker Registry (/DevOps/DevOps-ECS-ECR.php) Vagrant with Chef
CI/CD with CircleCI - Heroku deploy (/DevOps/DevOps- client provision
CircleCI-Heroku-Deploy.php) (/DevOps/Chef/Chef_Virtual_Machine_via_Vagrant_with_Chef_Client_Provision.ph
Introduction to Terraform with AWS elb & nginx
(/DevOps/DevOps-Terraform.php) Creating and using
Kubernetes I - Running Kubernetes Locally via Minikube cookbooks on a
(/DevOps/DevOps-Kubernetes-1-Running-Kubernetes-Locally- VirtualBox node
via-Minikube.php) (/DevOps/Chef/Chef_Creating_and_Using_Cookbooks_on_VirtualBox_Node.php)
Kubernetes II - kops on AWS (/DevOps/DevOps-
Kubernetes-II-kops-on-AWS.php) Chef server install
Kubernetes III - kubeadm on AWS (/DevOps/DevOps- on Ubuntu 14.04
Kubernetes-III-Kubernetes-on-Linux-with-kubeadm.php) (/DevOps/Chef/Chef_Server_install_on_EC2_ubuntu_14_04.php)

Chef workstation
setup on EC2
DEVOPS / SYS ADMIN Q & A Ubuntu 14.04
(/DevOps/Chef/Chef_Setting_Up_Workstation_on_EC2_Ubuntu_14.04.php)
(1A) - Linux Commands (/DevOps/DevOps-Sys-Admin-
Interview-Questions-Commands.php) Chef Client Node -
(1B) - Linux Commands (/DevOps/DevOps-Sys-Admin- Knife
Interview-Questions-Commands-2.php) Bootstrapping a
(2) - Networks (/DevOps/DevOps-Sys-Admin-Interview- node on EC2
Questions-Networks.php) ubuntu 14.04
(2B) - Networks (/DevOps/DevOps-Sys-Admin-Interview- (/DevOps/Chef/Chef_Client_Node_Knife_Bootstraping_a_Node_on_EC2_ubuntu_1
Questions-Networks-2.php)
(3) - Linux Systems (/DevOps/DevOps-Sys-Admin-Interview-
Questions-Systems.php)
(4) - Scripting (Ruby/Shell) (/DevOps/DevOps-Sys-Admin-
Interview-Questions-Scripting.php)
(5) - Con guration Management (/DevOps/DevOps-Sys-
Admin-Interview-Questions-Con gurations.php) Docker
(6) - AWS VPC setup (public/private subnets with NAT)
(/DevOps/DevOps-Sys-Admin-Interview-Questions-AWS-VPC-
Setup.php)
Docker install on
(6B) - AWS VPC Peering (/DevOps/DevOps-Sys-Admin-
Amazon Linux AMI
Interview-Questions-AWS-VPC-Peering.php)
(/DevOps/Docker/Docker_Install_On_Amazon_Linux_AMI.php)
(7) - Web server (/DevOps/DevOps-Sys-Admin-Interview-
Questions-Web-HTTP.php)
Docker install on
(8) - Database (/DevOps/DevOps-Sys-Admin-Interview-
EC2 Ubuntu 14.04
Questions-Database.php)
(/DevOps/Docker/Docker_Install_On_EC2_Ubuntu.php)
(9) - Linux System / Application Monitoring, Performance
Tuning, Pro ling Methods & Tools (/DevOps/DevOps-Sys-
Docker container
Admin-Interview-Questions-Linux-Monitoring-System-
vs Virtual Machine
Application-Performance-Tuning-Tools.php)
(/DevOps/Docker/Docker_Container_vs_Virtual_Machine.php)
(10) - Trouble Shooting: Load, Throughput, Response time
and Leaks (/DevOps/DevOps-Sys-Admin-Interview-Questions-
Docker install on
Trouble-Shooting-Slow-Application-Performance-BottleNecks-
Ubuntu 14.04
Leaks.php)
(/DevOps/Docker/Docker_Install_On_Ubuntu_14.php)
(11) - SSH key pairs & SSL Certi cate (/DevOps/DevOps-Sys-
Admin-Interview-Questions-SSH-Connection-SSL-
Certi cates.php) Docker Hello
(12) - Why is the database slow? (/DevOps/DevOps-Sys- World Application
Admin-Interview-Questions-Why-is-database-slow.php) (/DevOps/Docker/Docker_Hello_World_Application.php)
(13) - Is my web site down? (/DevOps/DevOps-Sys-Admin-
Interview-Questions-Is-Website-down.php) Nginx image -
(14) - Is my server down? (/DevOps/DevOps-Sys-Admin- share/copy les,
Interview-Questions-Is-Server-down.php) Docker le
(15) - Why is the server sluggish? (/DevOps/DevOps-Sys- (/DevOps/Docker/Docker_Nginx_WebServer.php)
Admin-Interview-Questions-Why-is-theServer-slow.php)
(16A) - Serving multiple domains using Virtual Hosts - Working with
Apache (/DevOps/DevOps-Sys-Admin-Interview-Questions- Docker images :
Serving-Multiple-Domains-Using-Virtual-Hosts-Apache.php) brief introdution
(16B) - Serving multiple domains using server block - Nginx (/DevOps/Docker/Docker_Working_with_images.php)
(/DevOps/DevOps-Sys-Admin-Interview-Questions-Serving-
Multiple-Domains-Using-Virtual-Hosts-Nginx.php) Docker image and
(16C) - Reverse proxy servers and load balancers - Nginx container via
(/DevOps/DevOps-Sys-Admin-Interview-Questions-Reverse- docker commands
proxy-servers-and-load-balancing-Nginx.php) (search, pull, run,
(17) - Linux startup process (/DevOps/DevOps-Sys-Admin- ps, restart, attach,
Interview-Questions-Linux-Boot-Startup-Process.php) and rm)
(18) - phpMyAdmin with Nginx virtual host as a subdomain (/DevOps/Docker/Docker_Commands_for_Images_Container.php)
(/DevOps/DevOps_phpMyAdmin_Nginx_Subdomain.php)
(19) - phpMyAdmin with Nginx virtual host as a subdomain More on docker
(/DevOps/DevOps_phpMyAdmin_Nginx_Subdomain.php) run command
(20) - Log Rotation (/DevOps/DevOps-Sys-Admin-Interview- (docker run -it,
Questions-Log-Rotation.php) docker run --rm,
(21) - Monitoring Metrics (/DevOps/DevOps-Sys-Admin- etc.)
Interview-Questions-Monitoring-Metrics.php) (/DevOps/Docker/Docker_Run_Command.php)
(22) - lsof (/DevOps/DevOps-Sys-Admin-Interview-
Questions-lsof.php) Docker Persistent
(23) - Wireshark introduction (/DevOps/DevOps-WireShark- Storage
Tutorial-Introduction.php) (/DevOps/Docker/Docker_Container_Persistent_Storage_Data_Share.php)
(24) - User account management (/DevOps/DevOps-Sys-
Admin-Interview-Questions-Linux-User-Account- File sharing
Management.php) between host and
(25) - Domain Name System (DNS) (/DevOps/DevOps-Sys- container (docker
Admin-Interview-Questions-DNS.php) run -d -p -v)
(0) - Linux Sys Admin's Day to Day tasks (/DevOps/DevOps- (/DevOps/Docker/Docker_File_Share_between_Host_and_Container.php)
Sys-Admin-Interview-Questions-Day-To-Day-Tasks.php)
Linking containers
and volume for
datastore
(/DevOps/Docker/Docker_Container_Linking_Connect_with_linking_system_Comm
Linux - system, cmds &
shell programming Docker le - Build
Docker images
1. Linux Tips - links, vmstats, rsync automatically I -
FROM,
(/Linux/linux_tips1.php)
MAINTAINER, and
2. Linux Tips 2 - ctrl a, curl r, tail -f, umask
build context
(/Linux/linux_tips2.php)
(/DevOps/Docker/Docker_Docker le_to_build_images_automatically.php)
3. Linux - bash I (/Linux/linux_tips2_bash.php)
4. Linux - bash II (/Linux/linux_bash_2.php) Docker le - Build
5. Linux - Uncompressing 7z le Docker images
(/Linux/linux_7z_uncompress.php) automatically II -
6. Linux - sed I (substitution: sed 's///', sed -i) revisiting FROM,
(/Linux/linux_sed_tutorial.php) MAINTAINER, build
context, and
7. Linux - sed II ( le spacing, numbering, text conversion
caching
and substitution) (/Linux/linux_sed_tutorial_2.php)
(/DevOps/Docker/Docker_Docker le_to_build_images_automatically_2.php)
8. Linux - sed III (selective printing of certain lines,
selective de nition of certain lines) Docker le - Build
(/Linux/linux_sed_tutorial_3.php) Docker images
9. Linux - 7 File types : Regular, Directory, Block le, automatically III -
Character device le, Pipe le, Symbolic link le, and RUN
Socket le (/Linux/linux_File_Types.php) (/DevOps/Docker/Docker_Docker le_to_build_images_automatically_3.php)

10. Linux shell programming - introduction

Docker le - Build
(/Linux/linux_shell_programming_tutorial.php)
Docker images
11. Linux shell programming - variables (readonly, unset) automatically IV -
(/Linux/linux_shell_programming_tutorial2_variables.php) CMD
12. Linux shell programming - special shell variables (/DevOps/Docker/Docker_Docker le_to_build_images_automatically_4_CMD.php
(/Linux/linux_shell_programming_tutorial3_special_variables.php)
13. Linux shell programming : arrays - three di erent Docker le - Build
ways of declaring arrays & looping with $*/$@ Docker images
automatically V -
(/Linux/linux_shell_programming_tutorial4_arrays.php)
WORKDIR, ENV,
14. Linux shell programming : operations on array ADD, and
ENTRYPOINT
(/Linux/linux_shell_programming_tutorial5_array_operations.php)
15. Linux shell programming : variables & commands (/DevOps/Docker/Docker_Docker le_to_build_images_automatically_5_WORKDIR

substitution
Docker Compose -
(/Linux/linux_shell_programming_tutorial6_shell_substitution.php)
A gentle
16. Linux shell programming : metacharacters & quotes introduction with
(/Linux/linux_shell_programming_tutorial7_metacharacters_quotes.php)
WordPress
17. Linux shell programming : input/output redirection & (/DevOps/Docker/Docker-
here document Compose.php)
(/Linux/linux_shell_programming_tutorial8_input_output_redirection.php)
18. Linux shell programming : loop control - for, while, MEAN Stack app
on Docker
break, and break n
containers : micro
(/Linux/linux_shell_programming_tutorial9_loop_control.php)
services (/MEAN-
19. Linux shell programming : for-loop Stack/MEAN-Stack-
(/Linux/linux_for_loop.php) NodeJS-Angular-
20. Linux shell programming : if/elif/else/ (/Linux/linux-If- Docker.php)
elif-else-statement.php)
21. Managing User Account - useradd, usermod, and MEAN Stack app
userdel on Docker
(/Linux/linux_user_account_useradd_usermod_userdel.php) containers : micro
services via
22. Linux Secure Shell (SSH) I : key generation, private key
docker-compose
and public key
(/MEAN-
(linux_Secure_Shell_SSH_Key_generation_public_private.php) Stack/MEAN-Stack-
23. Linux Secure Shell (SSH) II : ssh-agent & scp NodeJS-Angular-
(/Linux/linux_Secure_Shell_SSH_II_ssh_agent_and_scp.php) Docker-
24. Linux Secure Shell (SSH) III : SSH Tunnel as Proxy - Compose.php)
Dynamic Port Forwarding (SOCKS Proxy)
Docker Compose
(/Linux/linux_Secure_Shell_SSH_III_ssh_Tunnel_SOCKS_Proxy_Dynamic_Port_Forwarding.php)
with two
25. Linux Secure Shell (SSH) IV : Local port forwarding
containers - Flask
(outgoing ssh tunnel)
REST API service
(/Linux/linux_Secure_Shell_SSH_IV_ssh_Tunnel_Local_Port_Forwarding.php)
container and an
26. Linux Secure Shell (SSH) V : Reverse SSH Tunnel Apache server
(remote port forwarding / incoming ssh tunnel) /) container
(/DevOps/Docker/Docker-
(/Linux/linux_Secure_Shell_SSH_V_ssh_Reverse_SSH_Tunnel_Remote_Port_Forwarding.php)
27. Linux Processes and Signals Compose-
(/Linux/linux_process_and_signals.php) FlaskREST-Service-
Container-and-
28. Linux Drivers 1 (/Linux/linux_drivers_1.php)
Apache-
29. tcpdump (/Linux/tcpdump.php)
Container.php)
30. Linux Debugging using gdb
(/Linux/debugging_gdb.php) Docker compose :
31. Embedded Systems Programming I - Introduction Nginx reverse
(/cplusplus/embeddedSystemsProgramming.php) proxy with
32. Embedded Systems Programming II - gcc ARM multiple
Toolchain and Simple Code on Ubuntu/Fedora containers
(/DevOps/Docker/Docker-
(/cplusplus/embeddedSystemsProgramming_gnu_toolchain_ARM_cross_compiler.php)
Compose-Nginx-
33. LXC (Linux Container) Install and Run
Reverse-Proxy-
(/Linux/linux_LXC_Linux_Container_Install_Run.php)
Multiple-
34. Hadoop - 1. Setting up on Ubuntu for Single-Node Containers.php)
Cluster
(/Linux/hadoop_setting_up_on_ubuntu_single_node_cluster.php)
Docker Cheat
35. Hadoop - 2. Runing on Ubuntu for Single-Node Cluster Sheet
(/DevOps/Docker/Docker-
(/Linux/hadoop2_running_on_ubuntu_single_node_cluster.php)
36. ownCloud 7 install Cheat-Sheet.php)

(/Linux/linux_ubuntu_owncloud7_install.php)
Installing LAMP via
37. Ubuntu 14.04 guest on Mac OSX host using VirtualBox
puppet on Docker
I (/DevOps/Docker/Installing-
(/Linux/Ubuntu_Desktop_on_Mac_OSX_using_VirtualBox_4_3.php)
LAMP-with-
38. Ubuntu 14.04 guest on Mac OSX host using VirtualBox puppet-on-
II Docker.php)
(/Linux/Ubuntu_Desktop_on_Mac_OSX_using_VirtualBox_4_3_II.php)
39. Windows 8 guest on Mac OSX host using VirtualBox I Docker install via
Puppet
(/Linux/Windows_8_on_Mac_OSX_using_VirtualBox_4_3.php)
(/DevOps/Docker/Docker_puppet.php)
40. Ubuntu Package Management System (apt-get vs
dpkg) (/Linux/linux-Package-Management-System- Nginx Docker
Ubuntu-apt-get-vs-dpkg.php) install via Ansible
41. How to Make a Self-Signed SSL Certi cate (/DevOps/Ansible/Ansible-
(/Linux/Linux-How-to-Make-a-Self-Signed-SSL- Deploy-Nginx-to-
Certi cate.php) Docker.php)
42. DevOps / Sys Admin interview questions Apache Hadoop
(/DevOps/DevOps-Sys-Admin-Interview-Questions- CDH 5.8 Install
Commands.php) with QuickStarts
Docker
(/Hadoop/BigData_hadoop_CDH5.8_QuickStarts_Docker_Install.php)

Elasticsearch
search
engine,
Logstash,
and
Kibana

Elasticsearch,
search engine
(/Hadoop/ELK/ELK_Elastic_Search_Tutorial.php)

Logstash with
Elasticsearch
(/Hadoop/ELK/ELK_ElasticSearch_Logstash.php)

Logstash,
Elasticsearch, and
Kibana 4
(/Hadoop/ELK/ELK_ElasticSearch_Logstash_Kibana4.php)

Elasticsearch with
Redis broker and
Logstash Shipper
and Indexer
(/Hadoop/ELK/ELK_Logstash_Shipper_Logstash_Indexer_with_Redis_Broker.php)

Samples of ELK
architecture
(/Hadoop/ELK/ELK_Architecture_Samples_ElasticSearch_Logstash_Kibana.php)

Elasticsearch
indexing
performance
(/Hadoop/ELK/ELK_Elastic_Search_Tutorial_Indexing_Performance.php)

Vagrant

VirtualBox &
Vagrant install on
Ubuntu 14.04
(/DevOps/Vagrant/Vagrant_VirtualBox.php)

Creating a
VirtualBox using
Vagrant
(/DevOps/Vagrant/Creating_Virtual_Boxes_Using_Vagrant.php)

Provisioning
(/DevOps/Vagrant/Vagrant_Provisioning_Synced_Folders.php)

Networking - Port
Forwarding
(/DevOps/Vagrant/Vagrant_Networking_Port_Forwarding.php)

Vagrant Share
(/DevOps/Vagrant/Vagrant_Share.php)

Vagrant Rebuild &

Teardown
(/DevOps/Vagrant/Vagrant_Rebuild_Teardown.php)

Vagrant & Ansible

(/DevOps/Vagrant/Vagrant_Ansible.php)

AWS
(Amazon
Web
Services)

AWS : Creating a
snapshot (cloning
an image)
(/DevOps/AWS/aws_snapshot_ami_creation_image_clone_instance.php)

AWS : Attaching
Amazon EBS
volume to an
instance
(/DevOps/AWS/aws_attaching_Amazon_EBS_volume_to_instance.php)

AWS : Adding swap

space to an
attached volume
via mkswap and
swapon
(/DevOps/AWS/aws_adding_swap_space_to_attached_volume_via_mkswap_and_

AWS : Creating an
EC2 instance and
attaching Amazon
EBS volume to the
instance using
Python boto
module with User
data
(/DevOps/AWS/aws_creating_an_instance_and_attaching_Amazon_EBS_volume_t

AWS : Creating an
instance to a new
region by copying
an AMI
(/DevOps/AWS/Launching-
Instance-to-a-New-
Region-from-an-
AMI.php)

AWS : S3 (Simple
Storage Service) 1
(/DevOps/AWS/aws_S3_Simple_Storage_Service.php)

AWS : S3 (Simple
Storage Service) 2 -
Creating and
Deleting a Bucket
(/DevOps/AWS/aws_S3_Simple_Storage_Service2_Creating_and_Deleting.php)

AWS : S3 (Simple
Storage Service) 3 -
Bucket Versioning
(/DevOps/AWS/aws_S3_Simple_Storage_Service3_Bucket_Versioning.php)

AWS : S3 (Simple
Storage Service) 4 -
Uploading a large
le
(/DevOps/AWS/aws_S3_uploading_large_ le.php)

AWS : S3 (Simple
Storage Service) 5 -
Uploading
folders/ les
recursively
(/DevOps/AWS/aws-
uploading-
recursive-folder-
le.php)

AWS : S3 (Simple
Storage Service) 6 -
Bucket Policy for
File/Folder
View/Download
(/DevOps/AWS/aws-
S3-bucket-policy-
view-
download.php)

AWS : S3 (Simple
Storage Service) 7 -
How to Copy or
Move Objects from
one region to
another
(/DevOps/AWS/aws-
s3-Con gure-
Cross-Region-
Replication-
Source-and-
Destination-
Buckets-Owned-
by-the-Same-AWS-
Account-How-to-
Copy-or-Move-
Objects-from-one-
Region-to-
another.php)

AWS : S3 (Simple
Storage Service) 8 -
Archiving S3 Data
to Glacier
(/DevOps/AWS/aws-
S3-Simple-Storage-
Service-Archiving-
Amazon-S3-Data-
to-Amazon-
Glacier.php)

AWS : Creating a
CloudFront
distribution with
an Amazon S3
origin
(/DevOps/AWS/aws_Creating_CloudFront_Distribution_with_an_Amazon_S3_Orig

AWS : WAF (Web

Application
Firewall) with
precon gured
CloudFormation
template and Web
ACL for CloudFront
distribution
(/DevOps/AWS/aws-
WAF-Web-
Application-
Firewall.php)

AWS : CloudWatch
& Logs with
Lambda Function /
S3
(/DevOps/AWS/aws-
CloudWatch-logs-
Lambda-S3.php)

AWS : CLI
(Command Line
Interface)
(/DevOps/AWS/aws-
CLI-Command-
Line-Interface.php)

AWS : CLI (ECS with

ALB & autoscaling)
(/DevOps/AWS/aws-
Amazon-ECS-ALB-
Autoscaling-
CLI.php)

AWS Application
Load Balancer
(ALB) and ECS with
Flask app
(/DevOps/AWS/aws-
ELB-ALB-
Application-Load-
Balancer-ECS.php)

AWS : Load
Balancing with
HAProxy (High
Availability Proxy)
(/DevOps/AWS/aws-
Load-Balancing-
with-HAProxy-
High-Availability-
Proxy.php)

AWS : VirtualBox
on EC2
(/DevOps/AWS/aws_VirtualBox_On_EC2.php)

AWS : NTP setup

on EC2
(/DevOps/AWS/aws_NTP.php)

AWS & OpenSSL :

Creating /
Installing a Server
SSL Certi cate
(/DevOps/AWS/aws-
HTTPS-OpenSSL-
Certi cate.php)

AWS : OpenVPN
Access Server 2
Install
(/DevOps/AWS/aws-
OpenVPN-Access-
Server-Install.php)

AWS : VPC (Virtual

Private Cloud) 1 -
netmask, subnets,
default gateway,
and CIDR
(/DevOps/AWS/aws-
VPC-Virtual-
Private-Cloud-1-
netmast-subnet-
default-gateway-
CIDR.php)

AWS : VPC (Virtual

Private Cloud) 2 -
VPC Wizard
(/DevOps/AWS/aws-
VPC-Virtual-
Private-Cloud-2-
VPC-Wizard.php)

AWS : VPC (Virtual

Private Cloud) 3 -
VPC Wizard with
NAT
(/DevOps/AWS/aws-
VPC-Virtual-
Private-Cloud-3-
VPC-Wizard-with-
NAT.php)

DevOps / Sys
Admin Q & A (VI) -
AWS VPC setup
(public/private
subnets with NAT)
(/DevOps/DevOps-
Sys-Admin-
Interview-
Questions-AWS-
VPC-Setup.php)

AWS - OpenVPN
Protocols : PPTP,
L2TP/IPsec, and
OpenVPN
(/DevOps/AWS/aws-
VPN-Protocols-
OpenVPN-IPsec-
L2TP-PPTP.php)

AWS : Autoscaling
group (ASG)
(/DevOps/AWS/aws-
Autoscaling-
Group-ASG.php)

AWS : Adding a
SSH User Account
on Linux Instance
(/DevOps/AWS/aws-
adding-a-ssh-user-
account-on-linux-
instance.php)

AWS : Windows
Servers - Remote
Desktop
Connections using
RDP
(/DevOps/AWS/aws_Windows_Server_Remote_Desktop_Connections_Using_RDP

AWS : Scheduled
stopping and
starting an
instance - python
& cron
(/DevOps/AWS/aws_stopping_starting_instances.php)

AWS : Elastic
Beanstalk with
NodeJS
(/DevOps/AWS/aws-
Elastic-Beanstalk-
with-NodeJS.php)

AWS : Detecting
stopped instance
and sending an
alert email using
Mandrill smtp
(/DevOps/AWS/aws_detecting_stopped_instance_sending_email_using_mandrill_
AWS : Identity and
Access
Management (IAM)
Roles for Amazon
EC2
(/DevOps/AWS/aws-
IAM-Roles.php)

AWS : Identity and

Access
Management (IAM)
Policies
(/DevOps/AWS/aws-
IAM-Policies.php)

AWS : Creating IAM

Roles and
associating them
with EC2 Instances
in CloudFormation
(/DevOps/AWS/aws-
creating-IAM-
Roles-and-
associating-them-
with-EC2-
Instances-in-
CloudFormation.php)

AWS Identity and

Access
Management (IAM)
Roles, SSO(Single
Sign On),
SAML(Security
Assertion Markup
Language),
IdP(identity
provider),
STS(Security Token
Service), and
ADFS(Active
Directory
Federation
Services)
(/DevOps/AWS/aws-
IAM-Roles-SSO-
Single-Sign-On-
SAML-Security-
Assertion-Markup-
Language-ADFS-
Active-Directory-
Federation-
Services.php)

AWS : Amazon
Route 53
(/DevOps/AWS/aws-
Route53-DNS.php)

AWS : Amazon
Route 53 - DNS
(Domain Name
Server) setup
(/DevOps/AWS/aws-
Route53-DNS-
Domain-Name-
Server-Setup.php)

AWS : Amazon
Route 53 -
subdomain setup
and virtual host on
Nginx
(/DevOps/AWS/aws-
Route53-DNS-
Subdomain.php)
AWS Amazon
Route 53 : Private
Hosted Zone
(/DevOps/AWS/aws-
Route53-DNS-
Private-Hosted-
Zone.php)

AWS : SNS (Simple

Noti cation
Service) example
with ELB and
CloudWatch
(/DevOps/AWS/aws-
Amazon-SNS-
Simple-
Noti cation-
Service-example-
with-ELB.php)

AWS : Lambda with

AWS CloudTrail
(/DevOps/AWS/aws-
Lambda-with-AWS-
CloudTrail.php)

AWS : SQS (Simple

Queue Service)
with NodeJS and
AWS SDK
(/DevOps/AWS/aws-
Amazon-SQS-
Simple-Queue-
Service-with-
NodeJS-AWS-
SDK.php)

AWS : Redshift
data warehouse
(/DevOps/AWS/aws_Redshift_data_warehouse.php)

AWS :
CloudFormation -
templates, change
sets, and CLI
(/DevOps/AWS/aws-
CloudFormation-
Templates.php)

AWS :
CloudFormation
Bootstrap
UserData/Metadata
(/DevOps/AWS/aws-
CloudFormation-
Bootstrap-
UserData.php)

AWS :
CloudFormation -
Creating an ASG
with rolling update
(/DevOps/AWS/aws-
CloudFormation-
Autoscaling-
Group-ASG-
Application-Load-
Balancer-ALB-with-
Update-Policy-
Rolling-
Updates.php)

AWS : OpsWorks
(/DevOps/AWS/aws-
OpsWorks.php)

AWS CodeDeploy :
Deploy an
Application from
GitHub
(/DevOps/AWS/aws-
CodeDeploy-
Deploy-an-
Application-from-
GitHub.php)

AWS EC2
Container Service
(ECS)
(/DevOps/AWS/aws-
Amazon-EC2-
Container-Service-
ECS.php)

AWS Hello World

Lambda Function
(/DevOps/AWS/aws-
Hello-World-
Lambda-
Function.php)

AWS Node.js
Lambda Function
& API Gateway
(/DevOps/AWS/aws-
Lambda-Nodejs-
API-Gateway.php)

Amazon Kinesis
Streams
(/DevOps/AWS/aws-
Amazon-Kinesis-
Streams.php)

Amazon
DynamoDB
(/DevOps/AWS/aws-
Amazon-
DynamoDB.php)

Amazon ML
(Machine Learning)
(/DevOps/AWS/aws-
ML-Machine-
Learning.php)

Simple Systems
Manager (SSM)
(/DevOps/AWS/aws-
SSM.php)

AWS : RDS
Connecting to a DB
Instance Running
the SQL Server
Database Engine
(/DevOps/AWS/aws_Connecting_to_a_DB_Instance_Running_the_SQL_Server_Dat

AWS : RDS
Importing and
Exporting SQL
Server Data
(/DevOps/AWS/aws_RDS_Importing_and_Exporting_SQL_Server_Data.php)

AWS : RDS
PostgreSQL &
pgAdmin III
(/DevOps/AWS/aws_RDS_PostgreSQL_pgAdminIII.php)

AWS : RDS
PostgreSQL 2 -
Creating/Deleting
a Table
(/DevOps/AWS/aws_RDS_PostgreSQL_pgAdminIII_Creating_Table.php)
AWS : MySQL
Replication :
Master-slave
(/DevOps/AWS/aws-
MySQL-
Replication-
Master-Slave.php)

AWS : MySQL
backup & restore
(/DevOps/AWS/aws-
MySQL-Backup-
mysqldump-
Restore.php)

AWS RDS : Cross-

Region Read
Replicas for MySQL
and Snapshots for
PostgreSQL
(/DevOps/AWS/aws-
RDS-Cross-Region-
Read-Replicas-for-
MySQL-Snapshot-
for-
PostgreSQLs.php)

AWS : Restoring
Postgres on EC2
instance from S3
backup
(/DevOps/AWS/aws_S3_EC2_PostGres_Backup_Restore.php)

AWS : Scaling-Up
(/DevOps/AWS/aws-
Scaling-Up.php)

Big Data
&
Hadoop
Tutorials

Hadoop 2.6 -
Installing on
Ubuntu 14.04
(Single-Node
Cluster)
(/Hadoop/BigData_hadoop_Install_on_ubuntu_single_node_cluster.php)

Hadoop 2.6.5 -
Installing on
Ubuntu 16.04
(Single-Node
Cluster)
(/Hadoop/BigData_hadoop_Install_on_ubuntu_16_04_single_node_cluster.php)

Hadoop - Running
MapReduce Job
(/Hadoop/BigData_hadoop_Running_MapReduce_Job.php)

Hadoop -
Ecosystem
(/Hadoop/BigData_hadoop_Ecosystem.php)

CDH5.3 Install on
four EC2 instances
(1 Name node and
3 Datanodes) using
Cloudera Manager
5
(/Hadoop/BigData_hadoop_CDH5_Install.php)

CDH5 APIs
(/Hadoop/BigData_hadoop_CDH5_APIs.php)

QuickStart VMs for

CDH 5.3
(/Hadoop/BigData_hadoop_QuickStart
VMs for CDH
5.3.x.php)

QuickStart VMs for

CDH 5.3 II - Testing
with wordcount
(/Hadoop/BigData_hadoop_QuickStart
VMs for CDH
5.3.x_II.php)

QuickStart VMs for

CDH 5.3 II - Hive
DB query
(/Hadoop/BigData_hadoop_QuickStart%20VMs%20for%20CDH%205.3.x_Hive_Ta

Scheduled start
and stop CDH
services
(/Hadoop/BigData_hadoop_CDH5_stop_start_services.php)

CDH 5.8 Install

with QuickStarts
Docker
(/Hadoop/BigData_hadoop_CDH5.8_QuickStarts_Docker_Install.php)

Zookeeper & Kafka

Install
(/Hadoop/BigData_hadoop_Zookeeper_Kafka.php)

Zookeeper & Kafka

- single node single
broker
(/Hadoop/BigData_hadoop_Zookeeper_Kafka_single_node_single_broker_cluster

Zookeeper & Kafka

- Single node and
multiple brokers
(/Hadoop/BigData_hadoop_Zookeeper_Kafka_single_node_Multiple_broker_clust

OLTP vs OLAP
(/Hadoop/BigData_hadoop_OLTP_vs_OLAP.php)

Apache Hadoop
Tutorial I with CDH
- Overview
(/Hadoop/BigData_hadoop_Cloudera_Cluster_Tutorial_I.php)

Apache Hadoop
Tutorial II with
CDH - MapReduce
Word Count
(/Hadoop/BigData_hadoop_Tutorial_Cloudera_Cluster_Word_Count_MapReduce

Apache Hadoop
Tutorial III with
CDH - MapReduce
Word Count 2
(/Hadoop/BigData_hadoop_Tutorial_Cloudera_Cluster_Word_Count_MapReduce

Apache Hadoop
(CDH 5) Hive
Introduction
(/Hadoop/BigData_hadoop_CDH5_Hive_Introduction.php)

CDH5 - Hive
Upgrade to 1.3 to
from 1.2
(/Hadoop/BigData_hadoop_CDH5_Hive_Upgrade_2.php)

Apache Hive 2.1.0

install on Ubuntu
16.04
(/Hadoop/BigData_hadoop_Hive_Install_On_Ubuntu_16_04.php)

Apache HBase in
Pseudo-
Distributed mode
(/Hadoop/BigData_hadoop_HBase_Pseudo_Distributed.php)

Creating HBase
table with HBase
shell and HUE
(/Hadoop/BigData_hadoop_HBase_Creating_a_Table_with_Hbase_Shell_and_Hue

Apache Hadoop :
Hue 3.11 install on
Ubuntu 16.04
(/Hadoop/BigData_hadoop_Hue_Install_On_Ubuntu_16_04.php)

Creating HBase
table with Java API
(/Hadoop/BigData_hadoop_HBase_Table_with_Java_API.php)

HBase - Map,
Persistent, Sparse,
Sorted, Distributed
and
Multidimensional
(/Hadoop/BigData_hadoop_HBase_BigTable_Map_Persistent_Sparse_Sorted_Dist

Flume with CDH5:

a single-node
Flume deployment
(telnet example)
(/Hadoop/BigData_hadoop_CDH5_Flume_Introduction.php)

Apache Hadoop
(CDH 5) Flume with
VirtualBox : syslog
example via
NettyAvroRpcClient
(/Hadoop/BigData_hadoop_CDH5_Flume_CDH5_and_VirtualBox_NettyAvroRpcCl

List of Apache
Hadoop hdfs
commands
(/Hadoop/BigData_Hadoop_fs_commands_list.php)

Apache Hadoop :
Creating
Wordcount Java
Project with Eclipse
Part 1
(/Hadoop/BigData_hadoop_Creating_Java_Wordcount_Project_with_Eclipse_MapR

Apache Hadoop :
Creating
Wordcount Java
Project with Eclipse
Part 2
(/Hadoop/BigData_hadoop_Creating_Java_Wordcount_Project_with_Eclipse_MapR

Apache Hadoop :
Creating Card Java
Project with Eclipse
using Cloudera VM
UnoExample for
CDH5 - local run
(/Hadoop/BigData_hadoop_Creating_Java_Card_Project_with_Eclipse_MapReduce

Apache Hadoop :
Creating
Wordcount Maven