K8sGPT is a tool for scanning your Kubernetes clusters, diagnosing and triaging issues in simple english. It has SRE experience codified into its analyzers and helps to pull out the most relevant information to enrich it with AI.

K8SGPT is an advance AI algorithm analysis your cluster state and provide intelligent inside for troubleshooting.  

WorkFlow:

K8S is a dynamic network. Pods are ephemeral.  IP change on every restart.

Containers with in the pod shared a single network namespace.

K8S networking Model:

1) Every Pod receive a unique and cluster wide IP address.

2) All pods on the same node can communicate directly without NAT

3) All pods on different nods can communicate directly without NAT

4) A Pod self seen IP is identical to the IP other pods use to reach it [Flat network]

Kubernetes specifies what is required and CNI plugins decide How to implement it

Communication pattern in K8S

Container to Container - within same pod via loopbackup [127.0.0.1]

Pod to Pod - Direct IP communication across nodes without address translation

Pod to Service - Kube proxy intercepts traffic and load balancing to healthy end points

External to Service - Exposed via NodePort, LoadBalance type or Ingress controller

Node to Pod - Kubelet and monitoring agents

Infra [pause] container creates and own a network namespace for the pod. All application containers in the Pod share the Infra container namespace at startup.

Virtual [veth] pair : Two virtual NICs connect between Pod and Node side. One end lives inside the Pod's network namespace [eth0] and other end is attached to Node like linux bridge [cbr0]

Traffic flow : Pod [eth0] -> veth pair -> host bridge -> node routing table -> destination 

Analysis a packet flow under flannel CNI.

controlplane:~$ kubectl get pods -n kube-flannel -o wide

NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES

kube-flannel-ds-5sv5v   1/1     Running   0          15m    controlplane   <none>           <none>

kube-flannel-ds-n7dxx   1/1     Running   0          15m     node01         <none>           <none>

controlplane:~$ 

node01:~$ tcpdump -i flannel.1 -n 'tcp' -vvv

tcpdump: listening on flannel.1, link-type EN10MB (Ethernet), snapshot length 262144 bytes

^C

0 packets captured

0 packets received by filter

0 packets dropped by kernel

node01:~$ 

Service:

K8S will face very difficult to manage an IP address across PODs. This issue will fix by service which providing an stable virtual IP (Cluster IP). It will act as a load balancer across all the pods. It will enable a loose coupling within application.

Service components:

                             Deployment Strategies in Kubernetes

Kubernetes is a powerful container orchestration tool that simplifies the management of containerized applications. It handles provision, deployment, resource allocation, load balancing and Service discovery.

Kubernetes deployments are essentially just a wrapper around ReplicaSets. The ReplicaSet manages the number of running pods, and the Deployment implements features on top of that to allow rolling updates, health checks on pods, and easy roll-back of updates. During normal operations, the Deployment will just manage a single ReplicaSet which ensures that desired number of pods are running: When using Deployments, you should not directly manage the ReplicaSet that is created by the Deployment. All operations that you would perform on a ReplicaSet should be performed on the Deployment instead, which then manages the process for updating the ReplicaSet. 

Deployment strategies:

Rolling update Deployment : It will replace a old version of pod with new version one by one and reduce the downtime.

Recreate Deployment: It will bring down the old version and scale up with new version. It will require a downtime to perform this deployment.

Blue Green Deployment: It keeps one version alive while testing a new version and reduce a downtime/risk.

Canery Deployment: It allows gradual users to perform a testing with new version with subset of users before full rollout.

The most important option to configure rolling updates is the update strategy. In your 

Deployment manifest, spec.strategy.type has two possible values:

• RollingUpdate: New pods are added gradually, and old pods are terminated gradually

• Recreate: All old pods are terminated before any new pods are added.

Every request to Kubernetes is first authenticated, Authentication provides the identity of the caller issuing the request.  It could integrate deeply with pluggable authentication provide like Azure Active Directory to establish an identity within third party system.

Once user have been authenticated, the authorization phase determines whether they are authorized to perform the request. Authorization is a combination of identify the user and the resource and verb or action that user is trying to perform it.

Kubernetes makes distinction between user identity and service account identity. The service account identity is managed by K8S inside the cluster where as user identity is a normal user who is try to access the resource from outside of the K8S.

Kubernetes is supporting number of different authentication providers such as:

• The first one  is applied in the name space level
• The second one is applied through cluster level

ReplicaSets:

ReplicaSet acts as cluster wide Pod manager and ensuring that right types & number of Pods are running at all times.  It is providing the underpinning of self-healing for our applications at the infrastructure level.

 The reconciliation loop is the notion of desire state versus observed or current state.  The reconciliation loop is constantly running, observing the current level of the Pods and taking action in case of different state.

The ReplicaSet controller will notice that a Pod is missing and create a new copy, but still Pod is still running is available to developers for interactive debugging rather than debugging from logs.

ReplicaSet is define in the spec section while create a configuration file.

Ingress:

Kubernetes calls it HTTP based load balancing system Ingress. Ingress is a Kubernetes-native way to implement the "Virtual hosting" pattern.  The Kubernetes Ingress system works to simplify this by (a) standardizing that configuration (b) moving to standard Kubernetes object (c) merging multiple Ingress objects into a single config for the load balancer.

We have lot of Ingress controller in the market. I am going to use the contour [Ingress controller] along with Envoy load balancer.

Installed the contour as follows:

#kubectl apply -f https://j.hept.io/contour-deployment-rbac

 Labels & Annotation:

We can list the deployments along with labels:

#kubectl get deployments --show-labels

List the version with selector flag:

#kubectl get pods --selector="ver=2"

#kubectl get pods --selector="app in (test1,test2)"

Annotations provide a place to store additional metadata for the Kubernetes objects with the sole purpose of assisting tools and libraries. Labels are used to identify and group the objects whereas annotations are used to provide extra information about where an object came from, how to use it or policy around that object. Annotation used to track a rollout status and provide a necessary information require for rollback.

Annotation can be used to hold configuration data for the external tool such as third-party schedulers and configuration tools.

Services :

The DNS is a traditional system of service discovery on the Internet.

kubectl run is an easy way to create a deployment and kubectl expose is used to create a service.

ClusterIP:

A ClusterIP service is the default Kubernetes service. It gives you a service inside your cluster that other apps inside your cluster can access. There is no external access. This is a special IP address the system will load-balance across all of the Pods that are identified by the selector.

The YAML for a ClusterIP follow like this.

apiVersion: v1
kind: Service
metadata:
name: my-internal-service
spec:
selector:
app: my-app
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP

The customer request flow through as:

request -> proxy -> service -> Pod

Start the proxy to access the URL:

#kubectl proxy --port=8080

We can able to navigate this service as blow after enabled the proxy.

http://localhost:8080/api/v1/proxy/namespaces/default/services/my-internal-service:http/

P.S: We should not expose your service to the internet or any of production services.

NodePort:

A NodePort service is a most primitive way to get a external traffic directly to your service. NodePort as name implies open a specific port on all the nodes and traffic that sent to this port is forwarded to the service.

The Yaml file for NodePort setup as follows:

apiVersion: v1
kind: Service
metadata:
name: my-nodeport-service
spec:
selector:
app: my-app
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
nodePort: 30036
protocol: TCP

The main difference between NodePort and ClusterIP is allocated additional port called the NodePort that specifies which port to open on all nodes, It will take a random port if we are not specific in the configuration file.

LoadBalancer:

A LoadBalancer is a standard way to expose your application to the internet. 

Ingress:

It is not a type of service but it will act as smart router Infront of your application.

This will allow us to perform the path based and subdomain based routing into backend services.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
backend:
serviceName: other
servicePort: 8080
rules:
- host: foo.mydomain.com
http:
paths:
- backend:
serviceName: foo
servicePort: 8080
- host: mydomain.com
http:
paths:
- path: /bar/*
backend:
serviceName: bar
servicePort: 8080

Ingress is a most useful if you want to expose multiple services under the same IP address and all the services are use the same L7 protocol [http]. We can buy a single loadbalance and optimal use the services through Ingress with low cost.