Architecture – Multi Cloud Solutions

Architecture, Azure Cloud, Cloud, Development, DevOps, KnowHow

Azure Kubernetes Service (AKS) with Terraform deployment

AKS components

I assumed that you are familiar with the Kubernetes Cluster concepts (elementary level). Therefore I didn’t do any deep dive into the elementary components. The focus of this post is the following topics:

Azure-related Kubernetes components
Deploying AKS with Terraform

The control plane (Kubernetes core component)

It’s the core of the Kubernetes Cluster and doesn’t matter on which cloud provider platform you are provisioning a cluster. The main OS for AKS is Linux based.

Node pool (AKS component)

AKS has two types of Node pools:

System Node Pool: contains the nodes on which the control plane is running. For the control plane’s high availability is recommended to have at least 3 nodes in the System Node Pool.
User Node pool: contains the nodes on which my applications, APIs, APPs, or Services are running. This node pool can have one of the following host’s OSs.
- Linux
- Windows

An AKS Cluster can have both Windows and Linux -based User Node Pools in parallel. We can use nodeSelector in the YAML file to specify on which User Node Pool my application should be deployed. See more in the video below.

Note:
The importance is that all the nodes in a Node pool (doesn’t matter System or User) have the same VM size. Because we can specify one VM size for one Node Pool.

Node components

Each node in the Node Pool is a VM. Kubernetes uses the following components to orchestrate the nodes and pods that are running on the nodes.

Kubelet: manages deployments
Kube-Proxy: manages the nodes’ networking
Container runtime: up and run container images

This video walkthrough the AKS core concept and components and its implementation in Terraform.

The PowerPoint slides of the video are available here.

Shared slides: https://www.slideshare.net/parisamoosavinezhad/aks-components

GitHub: https://github.com/ParisaMousavi/enterprise-aks

For a nodeSelector sample code see the sample YAML file here: https://github.com/ParisaMousavi/solution-11-aks-apps/blob/main/sample-win/sample.yaml it’s a ASP.Net Application that will be deployed on windows node.

AKS security (service principal or managed identity)

AKS Cluster needs access to other Azure resources e.g. for autoscaling must be able to expand the VM Scale Set and assign an IP Address to the VM. Therefore the AKS Cluster needs Network Contributor RBAC Role.

Kubele needs to pull images from Azure Container Registry, therefore it needs AcrPull RBAC Role.

Only an identity can obtain a role. In Azure, we have two possibilities:

Associate a Service Principal to a Service (old solution in 2022) and give RBAC roles to the service principal.
Assign an identity to a service (new solution in 2022) and give RBAC roles to this identity. Here we have two types of identities:
- System Managed Identity: is created automatically and assigned to a service and is deleted when the service is deleted
- User Managed Identity: is created by the user and the user should assign it to a service and is not deleted when the service is deleted.

In this video, I have explained how to configure the Terraform implementation to assign the User Managed Identity to AKS Cluster and Kubelet. In addition, has been explained how to assign RBAC roles to them and which RBAC role for which purpose should be assigned.

The PowerPoint slides of the video are available here.

Shared slides: https://www.slideshare.net/parisamoosavinezhad/aks-scurity-cluster-kubelet-access-to-services

GitHub: https://github.com/ParisaMousavi/enterprise-aks/tree/2022.10.24

AKS operation (scaling and autoscaling)

Architecture, Industry

ITIL

IT Service/ Management

IT Service Provides value through outcomes to customers. It brings value to the company.
IT Service must lead to or provide lower costs and risks.

Stakeholders of IT services

People who have a stake in the value of a service.
Provider & customer of a service, who define system requirements and take responsibility for the outcomes of the service.

Value and outcomes

Value is the perceived benefit, usefulness, or importance of something to a stakeholder. The value of a service is what the customer seeks. The value has subjective quality.

The value definition is different in ITIL V3 and V4.

Outcome is the result for a stakeholder enabled by one or more outputs. Ex. the outcome of the haircut is that my hair looks gut.

Outputs are tangible or intangible deliverables. Ex. check if the haircut is complete.

All means that the service management capability must include systems for ensuring both gut reality and good perception over time and changing circumstances.

The ITIL is

A set of IT Service Management best practices
- About how to provide quality IT services, including the processes, functions, and other capabilities needed.
A whole industry of training and consulting companies and there’s a very popular set of certifications based on the publications.
ITIL is vendor-natural

References

Linkedin Learning

Architecture, AWS, Azure Cloud, Cloud, GCP

Clouds : Shared responsibility model

In doesn’t make difference which cloud vendor you have chosen as the platform. All of them follow the shared responsibility model.

What does it mean?

It means the cloud provider has the security responsibility of the cloud and cloud customer has the security responsibility in the cloud.

Azure	AWS	GCP	IBM
Shared responsibility model	Shared responsibility model	Shared responsibility model	Shared responsibility model

What is customer responsible for?

Configure the access to the resources e.g. servers
Responsible for operating system hardening of the servers
Ensure the disk volume has been encrypted
Determine the identity and access permissions of specific resources
ooo

Who should take care of security?

In companies where they up and run services/application on the cloud, the responsible teams have to have enough knowledge about the security on the cloud.

Developers and Enterprise architect	Ensure cloud services they use are designed and deployed with security.
DevOps and SRE Teams	Ensure security introduced into the infrastructure build pipeline and the environments remain secure post-production.
InfoSec Team	Secure systems

In which step of the project the security have to be applied?

Architecture, Azure Cloud, Industry

Gatekeeper pattern

Architecture

Lambda Architecture in Google & Azure Could

Lambda Architecture Definition

Lambda Architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch stream-processing methods to design a robust, scalable and fault-tolerance (human and machine) big data systems.

Lamba Architecture tries tries also balancing between the latency & Accuracy.

Lambda Architecture Layers

Master Layery

Serving Layer

Speed Layer

Lambda Architecture Properties:

A paradigm for Big Data
In data processing for balance on throughput , latency, fault-tolerance and scalable.
For modern data warehouse

Applying the Lambda Architecture with Spark, Kafka, and Cassandra

The toolings are the following:

Spark Data Frame & Spark SQL in addition to Spark’s Data Source API to load, store and manipulate data.
Spark Streaming & Spark-Kafka Integration techniques -> for reliability and speed
Develop a Kafka Data Producer -> to simulate the real-time data stream feed into streaming application.
Stateful Spark Streaming Application -> to preserve global state and use memory efficiently with approximate algorithms.
Errors & Code updates -> when we build a stateful Spark streaming application and a production application isn’t complete without the ability to handle errors and code updates.
Persist Data to Cassandra & HDFS -> for working with the scalable NoSQL database and persist the data to Cassandra and HDFS.

Note

Your Text Here

Lambda Architecture on Azure, Google and AWS

Azure	GCP	AWS

Related links

Azure Data Transfer and Transformation

References:

How to build a Big Data Pipeline