Financial Crimes Insight (FCI)

1.0.3+: From Containers to

Kubernetes Helm + Ingress
An overview of FCI infrastructure to help you troubleshoot problems

Sean Wingert
swinger@us.ibm.com

Copy of this presentation: https://ibm.biz/BdY5Le

FCI 1.0.3: From Containers to Helm + ingress
• Containers
• Docker
• Kubernetes
• Helm
• What is ingress?
• How do I troubleshoot FCI problems?
Preliminary

Financial Crimes Insight (FCI) uses frameworks too, including Docker,

Kubernetes, Helm, and nginx. Those frameworks are shared by all
verticals (a.k.a., “apps”) in FCI.
Preliminary

This TechTalk is designed to give you an overview of the underlying

frameworks to help you better understand and troubleshoot FCI.
Preliminary

More details about what I’ll cover are already on our YouTube channel:
https://ibm.biz/BdY5V5
 Preliminary

In Linux, it’s very helpful to use vim, not just vi, to troubleshoot, because you’ll be
working “remotely” (ssh commands):

Install vim like this:

Ubuntu: apt-get install -y vim
Red Hat Enterprise Linux ( RHEL)/CentOS: yum install –y vim

To use vim effectively, see:

https://www.youtube.com/watch?v=ggSyF1SVFr4 (8 min)
https://www.youtube.com/watch?v=Nim4_f5QUxA (59 min)
Preliminary

Books:
Preliminary

Problem:
Docker and especially Kubernetes and Helm are under heavy
development, so books often outdated—at least on specifics. They’re
useful for theory, but see the following Websites for the latest
commands.
Preliminary

Websites:
https://kubernetes.io/
https://github.com/helm/helm
https://www.docker.com/
 Preliminary

FCI Knowledge Center (KC):

https://ibm.biz/BdY5Jx

FCI Data Platform = Hadoop

FCI Platform = Kubernetes + Docker
 Preliminary
FCI Topology

https://www.ibm.com/support/knowledgecenter/SSCKRH_1.0.3/platform/install_deployment_topology.html
Preliminary
Containers
Containers

Like Virtual Machines (VMs), containers run your processes.

Containers

Containers are (basically) these 3 features of the Linux kernel:

cgroup
namespace
device mapper

https://fr.slideshare.net/jpetazzo/anatomy-of-a-container-namespaces-cgroups-some-filesystem-magic-linuxcon
Containers

Cgroups
limits how much / many system resources you can use
Containers

Namespaces:
Provide processes with their own
view of the system.

Namespaces = limits what you can

see (and therefore use)
Containers

Device Mapper:
The device mapper is a framework provided by the Linux kernel for
mapping physical block devices onto higher-level virtual block devices.
It forms the foundation of the logical volume manager (LVM), software
RAIDs and dm-crypt disk encryption, and offers additional features such
as file system snapshots.[1]
-- Wikipedia
Containers
Containers

VMs are sometimes called “pets” because you lavish time on them and
don’t want them to stop working.

Containers are often called “cattle” because they’re “ephemeral”—

meaning, when they stop, another starts to replace it. Their “state”
(stuff you save in them) is stored outside the container anyway, in NFS
in FCI, effectively making them shells.
Docker
Docker

Docker is an open-source engine that automates the deployment of

applications into containers.
James Turnbull – The Docker Book
Docker

In many ways, Docker is the new “internet darling.”

It became popular by making these Linux Kernel features easy to use
Docker

If you know somebody familiar with Berkley Standard Daemon (BSD),

say FreeBSD, ask them to explain Jails and you’ll see why Docker is so
friendly.
Docker

The purpose of Docker is to run containers.

Docker

What’s cool about containers is that they look and feel like VMs, but
they much faster and “lighter” because they share the hosts’ kernel,
unlike VMs.
Docker
Docker

We often say Docker when we mean Docker Engine: Linux container-

based runtime environment.
Docker

Images are the building

blocks of the Docker world.
You launch your containers
from images. Images are the
”build” part of Docker’s life
cycle.

All layers are read-only

except the top one.
Docker

Those layers depend on CoW = Copy on Write.

In short, you copy what you want to modify, modify it onto a new file
and update the old pointers to point to new locations.
Docker

Registry:
Docker hub (public) or your local one (private). Contains the repository.

Repository:
“Images are stored in collections, known as a repository, which is keyed by a name.” Effectively the
image with optional tags visible by curling the catalog

Catalog:
“The list of available repositories is made available through the catalog.”
 Docker

Querying the catalog

Docker

Docker Compose - which allows you to run stacks of containers to represent application
stacks, for example web server, application server and database server containers running
together to serve a specific application.

Docker Swarm - which allows you to create clusters of containers, called swarms, that
allow you to run scalable workloads.

NOTE: FCI doesn’t use the above in production but you’ll likely see those mentioned
elsewhere. The FCI team settled on Kubernetes instead of Docker Swarm because k8s has a
good mechanism to get two containers to talk to each other and because the tide is
moving towards Kubernetes.In other words Kubernetes won the “battle” over Swarm in
terms of running Docker Containers and is more popular than Swarm, despite that the
company that created Docker also created Swarm.
Docker

In the declarative model, you define the end state, and the system
determines how to get there.
Docker

In the imperative model, you define each step in sequence. You tell the
system how to get to the end state.
Docker

Docker uses the declarative model, notably with Swarm.

Docker

The FCI installer installs and configures all of these Docker and
Kubernetes components for you. Specifically it installs Docker CE on all
servers, all the rpms needed for Docker & Kubernetes, the Docker
Registry, Kubernetes, Helm, & Calico.
Docker

In FCI, if you already have Docker installed, the FCI installer will detect it
and utilize the existing Docker and lay Kubernetes on top of it.
Docker

Useful commands:
docker ps
shows a list of images on the current host
docker stats
shows CPU and disk usage on the current host
docker exec –ti CONTAINER_NAME /bin/bash
e.g., docker run –it ubuntu bash
docker version
gives the currently installed version of docker
Docker

Problems
What happens when you have 10, 50, 100, or 1,000 containers spread
across multiple hosts?
Docker alone can’t give you much insight about what they’re doing.
Docker

Sooner or later, you’ll want to manage all those containers.

Enter Kubernetes!
Kubernetes
Kubernetes

Kubernetes was designed to solve the problem of “managing many

containers" by “orchestrating” their lifecycle.
Kubernetes

"The first thing to know is that Kubernetes came out of Google. The
next thing to know is that in the summer of 2014 it was open-sourced
and handed over to the Cloud Native Computing Foundation (CNCF).
Since then it’s gone on to become one of the most important
container-related technologies in the world - probably second only to
Docker.“ - Turnbull
Kubernetes

History
Kubernetes was originally called Borg.
“Stories of Google crunching through billions of containers a week are
re-told at meetups all over the world.”
Kubernetes

Kubernetes is often abbreviated k8s

Kubernetes

Kubernetes was written in GoLang, which Google wrote.

Kubernetes

Kubernetes relies on YAML, a “superset of JSON”, for configuration.

Def: “YAML Ain’t Markup Language” (abbreviated YAML) is a data

serialization language designed to be human-friendly and work
well with modern programming languages for common everyday tasks.
Kubernetes

IBM uses K8s for many cloud-based suites of IBM software

For details, see the #armada-users Slack Channel
"Armada is the IBM Container Service on Kubernetes“
- Description in its Slack Channel
Kubernetes

ICP = IBM Cloud Private

“IBM Cloud Private is an application platform for developing and
managing on-premises, containerized applications. It is an integrated
environment for managing containers that includes the container
orchestrator Kubernetes, a private image repository, a management
console, and monitoring frameworks.”
 Kubernetes

In Kubernetes, A Node is a
bare-metal or Virtual
machine (VM), typically a
VM in FCI, that runs your
pods.

“First and foremost is the

kubelet. This is the main
Kubernetes agent that gets
installed on nodes. In fact
it’s fair to say that the
kubelet is the node. “
Kubernetes

Nodes are called Workers too.

The Master, AKA the Manager, is also called a Node.
Kubernetes

Kubernetes uses the term schedule to mean it starts running your

container on a given node.

Example: “Your pod was scheduled to node 2.”

Kubernetes

“At the highest level a Kubernetes Pod is a ring fenced environment to

run containers. The Pod itself doesn’t actually run anything, it’s just a
sandbox to run containers in. Keeping it high level, you ring-fence an
area of the host OS, build a network stack, create a bunch of kernel
namespaces, and run one or more containers in it - that’s a Pod.”
Kubernetes

Pods hold containers.

There are “regular” containers and “init” containers.

“An init container is a specialized Container that runs before app

Containers and can contain utilities or setup scripts not present in an
app image.”
https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
Kubernetes

To see both kinds of containers, issue:

kubectl describe pod PODNAME
Kubernetes

To get YAML output, issue this:

kubectl describe pod PODNAME -o yaml
Kubernetes

To get syntax highlighting, pipe the output to vim:

kubectl describe pod PODNAME -o yaml | vim -R -c “setf yaml” –
-R = open as Read-Only
-c = issue this command
- = read from Linux’s stdin (kubectl’s output essentially)
Kubernetes

When you issue:

kubectl get pods

NAME READY STATUS RESTARTS AGE

fcii-fci-insurance-liberty-6dcdbf968d-gl8vs 3/3 Running 0 6d
 Kubernetes

A Service in
Kubernetes is a
Resource and
REST object,
similar to a Pod.
Like all of the REST
objects, a Service
definition can be
POSTed to the
apiserver to create
a new instance.
Kubernetes

In Kubernetes, logs come from a container’s stdout

FCI uses sidecars to tail the containers’ logs to stdout so kubectl logs can read it.

stdout, stderr, and stdin:

Keeping with the Unix theme of "everything is a file," programs such
as ls actually send their results to a special file called standard output (often
expressed as stdout) and their status messages to another file called standard
error(stder) . By default, both standard output and standard error are linked
to the screen and not saved into a disk file.
In addition, many programs take input from a facility called standard
input (stdin) , which is, by default, attached to the keyboard. – Linux Command Line, William Shotts
Kubernetes

To access a specific container, mention it with -c like this:

kubectl logs fci-case-manager-fci-solution... -c fci-solution
 Kubernetes

https://sddevops.github.io/presentations/kubernetes-overview/
Kubernetes

To see logs for a Kubernetes pod, issue:

kubectl logs POD

Example:
kubectl logs fcii-fci-insurance-liberty-6dcdbf968d-gl8vs -c CONTAINERNAME
Kubernetes

Resources
pods
secrets
configmap
persistent volumes (PV)
persistent volume claims (PVC)
 Kubernetes

Like Docker, you can create a “one-off” container in a Kubernetes

cluster.
kubectl run -it tiny-tools --image=giantswarm/tiny-tools --restart=Never --rm -- dig google.com
Kubernetes

“The declarative model and the concept of desired state are two tings
at the very heart of the way Kubernetes works.”
Kubernetes

“If you are running Kubernetes worker and Kubernetes master on the
same server or virtual machine (VM), than you need to run the kubectl
taint command, which allows pods to be scheduled to run on the
Kubernetes master server.” -- IBM Knowledge Center
Kubernetes

When installing FCI, if you don't have a dedicated VM for Linux’s

Network File System (NFS), which works like Windows network shares,
if you're familiar with that, then we create an NFS server on the
Kubernetes Master.
Kubernetes

Namespaces

kubectl get pods –n kube-system

OR (longer):
kubectl get pods --namespace=kube-system
Kubernetes

Commands
kubectl ...
kubectl get pods
kubectl get pods -o wide
kubectl get pods -o yaml
kubectl get all -o wide
kubectl version
Kubernetes

"During the startup of a Pod, the Init Containers are started in order,
after the network and volumes are initialized. Each Container must exit
successfully before the next is started. If a Container fails to start due to
the runtime or exits with failure, it is retried..."
Kubernetes

"A Pod cannot be Ready until all Init Containers have succeeded...A Pod
that is initializing is in the Pending state but should have a condition
Initializing set to true.“

Useful command
watch kubectl get pods
Kubernetes

Stateful Sets are a k8s resource for multiple backend (databases).

“StatefulSet is the workload API object used to manage stateful
applications.” - source
Kubernetes

Problems
With Kubernetes, you only interact with one resource at a time.

If you have 10 resources, which is reasonable, you have to issue 10 commands:

kubectl apply -f pv.yaml
kubectl apply -f pv.yaml
kubectl apply -f service.yaml
...
Helm
Helm

Designed to solve the problem of managing multiple resources that

belong together. Instead of issuing 10 kubectl apply commands, you
issue 1 helm command.

"Helm is a tool for managing Kubernetes charts.”

Helm

Note: FCI’s installer manages Helm for you and customers, but it’s still useful
to know for troubleshooting.

cat /root/fci-install-toolkit-1.1/helm/install-20181113-160413.log
helm upgrade --install --recreate-pods fcii -f fcii-values.yaml --set
global.coreReleaseName=fci --set
global.initPvImage=fciicluster1.fyre.ibm.com:5000/centos:7 --set
global.managerFQDN=fciicluster1.fyre.ibm.com --set
global.nfsServer=fciicluster1.fyre.ibm.com --set
global.dockerRepository=fciicluster1.fyre.ibm.com:5000 /root/fcii-install-toolkit-
3.1/helm/fci-insurance-3.1.tgz
Helm

If you need to modify a Kubernetes resource, you can use kubectl edit
Potentially, that means 10 different kubectl edit commands.
If you use Helm to make those changes instead, you get rollbacks and
audits of changes. In seconds, you can undo mistakes.
Helm

In Helm, Charts are packages of pre-configured Kubernetes resources.

A Chart is a Helm package. It contains all of the resource definitions
necessary to run an application, tool, or service inside of a Kubernetes
cluster. Think of it like the Kubernetes equivalent of a Homebrew
formula, an Apt dpkg, or a Yum RPM file.
Helm

"A Helm Repository is the place where charts can be collected and
shared. It’s like Perl’s CPAN archive or the Fedora Package Database,
but for Kubernetes packages"
Helm

"A Helm Release is an instance of a chart running in a Kubernetes

cluster. One chart can often be installed many times into the same
cluster. And each time it is installed, a new release is created. Consider
a MySQL chart. If you want two databases running in your cluster, you
can install that chart twice. Each one will have its own release, which
will in turn have its own release name."
Helm

Helm has 2 parts:

Client: helm
Server/engine: called Tiller
Helm

Helm uses the Go templating language to allow end users to modify

configuration settings before deploying a chart.
Helm

Helm lets you version your changes through Rollbacks.

You can install a chart, upgrade to the next version (APAR), and track
which version is deployed.
Helm

Useful commands (for learning):

helm ls
helm get chart
helm get chart | vim -R -c "setf yaml" –
helm search mysql
helm rollback happy-panda 1
Ingress
Ingress

In FCI, Calico handles networking between VMs—i.e., inside the cluster.

“Project Calico is a layer 3-based networking model that uses the built-in
routing functions of the Linux kernel. Routes are propagated to virtual
routers on each host via Border Gateway Protocol (BGP). Calico can be used
for anything from small-scale deploys to large Internet-scale installations.
Because it works at a lower level on the network stack, there is no need for
additional NAT, tunneling, or overlays.”
Ingress

Getting traffic into the Kubernetes cluster is called “ingress.”

FCI uses a Helm chart to control ingress through nginx ingress

controller.

Fernando Diaz, an IBMer, has made many public contributions to that

project, as explained on its GitHub project page and video.
Ingress
Troubleshooting
Troubleshooting

Question:
How do I know if my system installed correctly?
Troubleshooting

Answer:

Check the installation log:

/root/fci-install-toolkit-1.1/helm/install-20181113-160413.log
Troubleshooting

Question:
How do I know if my system is running correctly?
Troubleshooting

Answer:
kubectl get all -o wide
Troubleshooting

Question:
What does this mean?

Error from server (BadRequest): a container name must be specified for

pod fcii-fci-insurance-liberty-6dcdbf968d-gl8vs, choose one of: [liberty
message-log trace-log]
Troubleshooting

Answer:
Mention the container with -c:

kubectl logs fcii-fci-insurance-liberty-6dcdbf968d-gl8vs –c trace-log