[{"content":"Running openai/gpt-oss-20b local model with NVIDIA L4 GPU this model can actually run on a consumer RTX Series GPU with ~16GB of VRAM. I divided it into two parts: running manually and using a container using the Ubuntu 24.04 LTS operating system.\nPreparation Installing drivers and dependencies\nwget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb \u0026amp;\u0026amp; rm -rf cuda-keyring_1.1-1_all.deb sudo apt update \u0026amp;\u0026amp; sudo apt install -y \\ linux-headers-$(uname -r) \\ libnvidia-compute-580 nvidia-dkms-580-open \\ datacenter-gpu-manager-4-cuda-all \\ datacenter-gpu-manager-exporter \\ cuda-toolkit nvtop build-essential We need a host reboot to apply the GPU driver.\nDriver Validation nvidia-smi nvidia-smi -L Running vLLM Manually Installing git LFS and clone openai/gpt-oss-20b repository\ncurl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash apt install git-lfs \u0026amp;\u0026amp; git lfs install git clone https://huggingface.co/openai/gpt-oss-20b /models/gpt-oss-20b Installing UV for python virtual environment\ncurl -LsSf https://astral.sh/uv/install.sh | sh cd /models \u0026amp;\u0026amp; uv venv source .venv/bin/activate uv pip install vllm==0.10.2 --torch-backend=auto Start serving the model with vllm using the following command\nexport VLLM_LOGGING_LEVEL=ERROR python3 -m vllm.entrypoints.openai.api_server \\ --host 0.0.0.0 --port 8000 \\ --api-key $API_TOKEN \\ --served-model-name openai/gpt-oss-20b \\ --model /models/gpt-oss-20b \\ --gpu-memory-utilization 0.90 \\ --chat-template-content-format openai \\ --tool-call-parser openai \\ --enable-auto-tool-choice \\ --trust-remote-code \\ --async-scheduling \\ --disable-log-requests You can monitor GPU utilization in real time using the nvtop command.\nWhen the model is successfully loaded, a message stating “application startup complete” will appear, and you will be able to use the model. Running vLLM with Container Installing Nvidia Container Toolkit\ncurl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo \\ gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \\ \u0026amp;\u0026amp; curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \\ sed \u0026#39;s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g\u0026#39; | \\ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.18.0-1 sudo apt update \u0026amp;\u0026amp; apt-get install -y \\ nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \\ nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \\ libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \\ libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION} Installing Docker and setup Nvidia Runtime\ncurl -fsSL https://get.docker.com | bash nvidia-ctk runtime configure --runtime=docker systemctl restart docker.service docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi If the GPU has been detected at the container level, proceed to the next step.\nCreate docker-compose.yaml file configuration\nservices: vllm: image: vllm/vllm-openai:v0.12.0 container_name: vllm-gpt-oss-20b restart: unless-stopped ports: - \u0026#34;8000:8000\u0026#34; environment: API_TOKEN: ${API_TOKEN} VLLM_LOGGING_LEVEL: ERROR volumes: - /models:/models command: \u0026gt; --host 0.0.0.0 --port 8000 --api-key ${API_TOKEN} --served-model-name openai/gpt-oss-20b --model /models/gpt-oss-20b --gpu-memory-utilization 0.90 --chat-template-content-format openai --tool-call-parser openai --enable-auto-tool-choice --trust-remote-code --async-scheduling --disable-log-requests deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] ipc: host ipc=host makes your container share memory space with the host machine, which is critical for running large models efficiently.\nrun it with the command docker compose up -d then monitor the log with the command docker compose logs -f.\nResult:\nUsing The Model Check the model with the GET API endpoint /v1/models as in the following example\ncurl -X GET http://127.0.0.1:8000/v1/models \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -H \u0026#34;Authorization: Bearer $API_TOKEN\u0026#34; | jq -r Result:\nThen perform a prompt test with POST to the API endpoint /v1/chat/completions as shown in the following example\ncurl -X POST http://127.0.0.1:8000/v1/chat/completions \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -H \u0026#34;Authorization: Bearer $API_TOKEN\u0026#34; \\ -d \u0026#39;{ \u0026#34;model\u0026#34;: \u0026#34;openai/gpt-oss-20b\u0026#34;, \u0026#34;messages\u0026#34;: [ { \u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;what is AI ?\u0026#34; } ], \u0026#34;temperature\u0026#34;: 0.3, \u0026#34;max_tokens\u0026#34;: 256, \u0026#34;stream\u0026#34;: false, \u0026#34;reasoning_effort\u0026#34;: \u0026#34;high\u0026#34; }\u0026#39; | jq -r Result:\nBenchmark Thanks to Yoosu-L for the easy-to-use llm benchmark tool.\ncurl -LO https://github.com/Yoosu-L/llmapibenchmark/releases/download/v1.0.7/llmapibenchmark_linux_amd64.tar.gz tar xzvf llmapibenchmark_linux_amd64.tar.gz We\u0026rsquo;ll test with up to 1,024 concurrency with run the following command:\n./llmapibenchmark_linux_amd64 \\ --base-url http://127.0.0.1:8000/v1 \\ --api-key $API_TOKEN \\ --model openai/gpt-oss-20b \\ --concurrency 1,2,4,8,16,32,64,128,256,512,1024 \\ --max-tokens 512 \\ --num-words 512 \\ --prompt \u0026#34;what is AI ?\u0026#34; \\ --format yaml | tee benchmark-results-gpt-oss-20b-nvidia-l4.yaml Results:\nAfter tested openai/gpt-oss-20b under different levels of concurrency to understand how it behaves as load increases. As concurrency goes up, the model clearly becomes faster at generating tokens overall, showing strong scaling behavior. Prompt throughput also improves with higher concurrency, especially in the mid range.\nWhile the fastest responses (min TTFT) stay consistently low, the slowest responses (max TTFT) increase noticeably as concurrency grows. This means most requests are still served quickly, but a small portion start to experience longer waits when the system is heavily loaded. Importantly, all requests succeed, even at the highest concurrency levels.\nBenchmark data can be found here\nReferences https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/ubuntu.html https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html https://cookbook.openai.com/articles/gpt-oss/run-vllm https://docs.astral.sh/uv/getting-started/installation https://docs.redhat.com/en/documentation/red_hat_ai_inference_server https://github.com/Yoosu-L/llmapibenchmark ","permalink":"http://vikipranata.is-a.dev/posts/vllm-gpt-oss-20b/","summary":"\u003cp\u003eRunning \u003ca href=\"https://huggingface.co/openai/gpt-oss-20b\"\u003eopenai/gpt-oss-20b\u003c/a\u003e local model with \u003ccode\u003eNVIDIA L4 GPU\u003c/code\u003e this model can actually run on a consumer RTX Series GPU with ~16GB of VRAM. I divided it into two parts: running manually and using a container using the \u003ccode\u003eUbuntu 24.04 LTS\u003c/code\u003e operating system.\u003c/p\u003e\n\u003ch2 id=\"preparation\"\u003ePreparation\u003c/h2\u003e\n\u003cp\u003eInstalling drivers and dependencies\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ewget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003esudo dpkg -i cuda-keyring_1.1-1_all.deb \u003cspan class=\"o\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e rm -rf cuda-keyring_1.1-1_all.deb\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003esudo apt update \u003cspan class=\"o\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e sudo apt install -y \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"se\"\u003e\u003c/span\u003e     linux-headers-\u003cspan class=\"k\"\u003e$(\u003c/span\u003euname -r\u003cspan class=\"k\"\u003e)\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"se\"\u003e\u003c/span\u003e     libnvidia-compute-580 nvidia-dkms-580-open \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"se\"\u003e\u003c/span\u003e     datacenter-gpu-manager-4-cuda-all \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"se\"\u003e\u003c/span\u003e     datacenter-gpu-manager-exporter \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"se\"\u003e\u003c/span\u003e     cuda-toolkit nvtop build-essential\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cblockquote\u003e\n\u003cp\u003eWe need a host reboot to apply the GPU driver.\u003c/p\u003e","title":"Running LLM OpenAI Open Source Model with vLLM and GPU NVIDIA L4"},{"content":"Preparation I used three VM nodes for this project with 8 Cores 8GB Memory and 80GB for the containers storage with operating systems Rocky Linux 9.6 with RKE2 v1.32.5+rke2r1 and Cilium v1.17.3\nNode Hostname vCPU Memory Storage PrivateNet Node Roles knode01master01 8 Core 8GB 80GB 172.16.0.211 Control-plane knode01master02 8 Core 8GB 80GB 172.16.0.212 Control-plane knode01master03 8 Core 8GB 80GB 172.16.0.213 Control-plane knode01worker01 8 Core 8GB 80GB 172.16.0.211 Worker knode01worker02 8 Core 8GB 80GB 172.16.0.212 Worker knode01worker03 8 Core 8GB 80GB 172.16.0.213 Worker All operations use the root user, be careful when running commands!\nIn this step execution on all nodes\nSet static hostname Set Up environment\nNODE_01=knode01master01 NODE_02=knode01master02 NODE_03=knode01master03 NODE_04=knode01worker01 NODE_05=knode01worker02 NODE_06=knode01worker03 NODE_IP01=172.16.0.211 NODE_IP02=172.16.0.212 NODE_IP03=172.16.0.213 NODE_IP04=172.16.0.214 NODE_IP05=172.16.0.215 NODE_IP06=172.16.0.216 then add static hostname in /etc/hosts for all nodes.\ncat \u0026lt;\u0026lt;EOF \u0026gt;\u0026gt; /etc/hosts $NODE_IP01 $NODE_01 $NODE_IP02 $NODE_02 $NODE_IP03 $NODE_03 $NODE_IP04 $NODE_04 $NODE_IP05 $NODE_05 $NODE_IP06 $NODE_06 EOF Installing packages \u0026amp; dependencies dnf install -y epel-release dnf install bash-color-prompt bash-completion \\ nano htop iftop iotop net-tools dnsutils \\ wget curl jq traceroute mtr tcpdump nmap setroubleshoot policycoreutils Create directory configurations\nmkdir -p /etc/rancher/rke2 Cluster Initialization In this step execution on all master nodes\ncat \u0026lt;\u0026lt;EOF | tee /etc/rancher/rke2/config.yaml node-ip: $(hostname -I | awk \u0026#39;{print $1}\u0026#39;) write-kubeconfig-mode: \u0026#34;0600\u0026#34; tls-san: - \u0026#34;localhost\u0026#34; - \u0026#34;127.0.0.1\u0026#34; cluster-cidr: 10.42.0.0/16 service-cidr: 10.43.0.0/16 # Make a etcd snapshot every 6 hours etcd-snapshot-schedule-cron: \u0026#34; */6 * * *\u0026#34; # Keep 56 etcd snapshorts (equals to 2 weeks with 6 a day) etcd-snapshot-retention: 56 enable-servicelb: false disable-kube-proxy: true disable: - rke2-ingress cni: cilium selinux: true EOF Download RKE2 from official repository curl -sfL https://get.rke2.io | \\ INSTALL_RKE2_CHANNEL=\u0026#34;v1.32.5+rke2r1\u0026#34; \\ RKE2_NODE_NAME=$(hostname -s) \\ INSTALL_RKE2_TYPE=\u0026#34;server\u0026#34; \\ sh - Set of Cilium Configuration Create cilium helm values file\ncat \u0026lt;\u0026lt;EOF | tee /var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml apiVersion: helm.cattle.io/v1 kind: HelmChartConfig metadata: name: rke2-cilium namespace: kube-system spec: valuesContent: |- k8sServiceHost: 127.0.0.1 k8sServicePort: 6443 kubeProxyReplacement: true cni: chainingMode: portmap localRedirectPolicy: true bandwidthManager: enabled: true bbr: true l2announcements: enabled: true leaseDuration: 15s leaseRenewDeadline: 5s easeRetryPeriod: 3s bpf: hostLegacyRouting: false masquerade: true hubble: enabled: false ipam: mode: kubernetes nodeIPAM: enabled: true enableLBIPAM: true EOF For now, we are disabling Hubble to minimize resource usage.\nEnable Node Local DNS Cache NodeLocal DNS Cache with Cilium in kube-proxy replacement mode\ncat \u0026lt;\u0026lt;EOF | tee /var/lib/rancher/rke2/server/manifests/rke2-coredns-config.yaml apiVersion: helm.cattle.io/v1 kind: HelmChartConfig metadata: name: rke2-coredns namespace: kube-system spec: valuesContent: |- nodelocal: enabled: true use_cilium_lrp: true EOF If all preparation is ready now we start and make service rke2-server to make running at system boot.\nsystemctl enable --now rke2-server.service You can troubleshoot/debug with journalctl -xefu rke2-server\nJoin another control-plane nodes get token join from first master node in cat /var/lib/rancher/rke2/server/agent-token\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/rancher/rke2/config.yaml node-ip: $(hostname -I | awk \u0026#39;{print $1}\u0026#39;) server: https://$NODE_IP01:9345 token: K10e6528a04f97734c36d94c9b70e699d5f1df3b254c69c2e91bfa896fa03b468c5::server:aae38fe9abdb13aacf5754e4ba40574c selinux: true EOF Now we start and make service rke2-server to make running at system boot.\nsystemctl enable --now rke2-server.service You can troubleshoot/debug with journalctl -xefu rke2-server\nJoin another worker nodes Download RKE2 from official repository\ncurl -sfL https://get.rke2.io | \\ INSTALL_RKE2_CHANNEL=\u0026#34;v1.32.5+rke2r1\u0026#34; \\ RKE2_NODE_NAME=$(hostname -s) \\ INSTALL_RKE2_TYPE=\u0026#34;agent\u0026#34; \\ sh - get token join from first master node in cat /var/lib/rancher/rke2/server/agent-token\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/rancher/rke2/config.yaml node-ip: $(hostname -I | awk \u0026#39;{print $1}\u0026#39;) server: https://$NODE_IP01:9345 token: K10e6528a04f97734c36d94c9b70e699d5f1df3b254c69c2e91bfa896fa03b468c5::server:aae38fe9abdb13aacf5754e4ba40574c selinux: true EOF Now we start and make service rke2-agent to make running at system boot.\nsystemctl enable --now rke2-agent.service You can troubleshoot/debug with journalctl -xefu rke2-agent\nAccessing the cluster and installing the utility Create .kube in home directory and copy or link cluster credentials and also set containerd endpoint in /etc/crictl.yaml file.\nmkdir ~/.kube ln -s /etc/rancher/rke2/rke2.yaml ~/.kube/config ln -s /var/lib/rancher/rke2/agent/etc/crictl.yaml /etc/crictl.yaml Now installing kubectl, crictl, helm, and cilium binary files.\ninstall /var/lib/rancher/rke2/bin/kubectl /usr/bin/kubectl install /var/lib/rancher/rke2/bin/crictl /usr/bin/crictl HELM_VER=$(curl -sL https://get.helm.sh/helm-latest-version) HELM_FILE=helm-$HELM_VER-linux-amd64.tar.gz curl -sSL https://get.helm.sh/helm-$HELM_VER-linux-amd64.tar.gz -o $HELM_FILE; tar -zxf $HELM_FILE install linux-amd64/helm /usr/bin/helm CILIUM_CLI_VER=$(curl -sL https://raw.githubusercontent.com/cilium/cilium-cli/refs/heads/main/stable.txt) curl -sSL https://github.com/cilium/cilium-cli/releases/download/$CILIUM_CLI_VER/cilium-linux-amd64.tar.gz -O; tar -zxf cilium-linux-amd64.tar.gz install cilium /usr/bin/cilium Set up completion bash to simplify administration.\nfor i in kubectl crictl helm cilium; do $i completion bash | tee /etc/bash_completion.d/$i \u0026gt; /dev/null; done source /usr/share/bash-completion/bash_completion All nodes have joined the cluster, you can see with kubectl get node,pod -A -o wide commands. Cilium CNI status. References https://docs.rke2.io/reference/server_config https://docs.rke2.io/reference/linux_agent_config https://docs.cilium.io/en/stable/installation/cni-chaining-portmap https://docs.cilium.io/en/stable/network/l2-announcements/ https://docs.cilium.io/en/stable/network/node-ipam/ https://docs.cilium.io/en/stable/network/lb-ipam/ https://docs.cilium.io/en/stable/network/kubernetes/bandwidth-manager/ https://docs.cilium.io/en/stable/network/kubernetes/local-redirect-policy/#node-local-dns-cache https://cilium.io/blog/2020/06/22/cilium-18/#kubeproxy-removal ","permalink":"http://vikipranata.is-a.dev/posts/rke2-with-cillium-cni/","summary":"\u003ch2 id=\"preparation\"\u003e\u003cem\u003ePreparation\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eI used three VM nodes for this project with 8 Cores 8GB Memory and 80GB for the containers storage with operating systems \u003cem\u003eRocky Linux 9.6\u003c/em\u003e with RKE2 v1.32.5+rke2r1 and Cilium v1.17.3\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e\u003cem\u003eNode Hostname\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003evCPU\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eMemory\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eStorage\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003ePrivateNet\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eNode Roles\u003c/em\u003e\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eknode01master01\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e80GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.211\u003c/td\u003e\n          \u003ctd\u003eControl-plane\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eknode01master02\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e80GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.212\u003c/td\u003e\n          \u003ctd\u003eControl-plane\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eknode01master03\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e80GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.213\u003c/td\u003e\n          \u003ctd\u003eControl-plane\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eknode01worker01\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e80GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.211\u003c/td\u003e\n          \u003ctd\u003eWorker\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eknode01worker02\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e80GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.212\u003c/td\u003e\n          \u003ctd\u003eWorker\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eknode01worker03\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e80GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.213\u003c/td\u003e\n          \u003ctd\u003eWorker\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cp\u003eAll operations use the \u003ccode\u003eroot\u003c/code\u003e user, be careful when running commands!\u003cbr\u003e\nIn this step execution on all nodes\u003c/p\u003e","title":"Kubernetes Cluster RKE2 with Cilium eBPF CNI"},{"content":"Preparation I used three VM nodes for this home lab project with 4 Cores 4GB Memory and 20GB for the containers storage with operating systems Rocky Linux 9.5 with kubernetes v1.32.5+k3s1 and cri-o v1.32.\nNode Hostname vCPU Memory Storage PrivateNet litekube01 4 Core 4GB 20GB 172.16.0.111 litekube02 4 Core 4GB 20GB 172.16.0.112 litekube03 4 Core 4GB 20GB 172.16.0.113 All operations use the root user, be careful when running commands!\nIn this step execution on all nodes\nOptional: Set SElinux to permissive mode for dev environments\nsetenforce 0 sed -i \u0026#39;s/^SELINUX=enforcing$/SELINUX=permissive/\u0026#39; /etc/selinux/config Set static hostname Set Up environment\nNODE_01=litekube01 NODE_02=litekube02 NODE_03=litekube03 NODE_IP01=172.16.0.111 NODE_IP02=172.16.0.112 NODE_IP03=172.16.0.113 then add static hostname in /etc/hosts for all nodes.\ncat \u0026lt;\u0026lt;EOF \u0026gt;\u0026gt; /etc/hosts $NODE_IP01 $NODE_01 $NODE_IP02 $NODE_02 $NODE_IP03 $NODE_03 EOF Add Repository and Install CRI-O CRIO_VERSION=v1.32 cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/yum.repos.d/cri-o.repo [cri-o] name=CRI-O baseurl=https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/rpm/ enabled=1 gpgcheck=1 gpgkey=https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/rpm/repodata/repomd.xml.key EOF Install packages \u0026amp; dependencies dnf install -y epel-release dnf install -y container-selinux cri-o Download and install k3s\ncurl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_CHANNEL=v1.32.5+k3s1 sh -s - server Bootsraping cluster Execute only in first control-plane node\nCreate folder /etc/rancher/k3s then create configuration file based on the environmental variables we defined earlier\nmkdir -p /etc/rancher/k3s cat \u0026lt;\u0026lt;EOF | tee /etc/rancher/k3s/config.yaml cluster-init: true node-ip: $(hostname -I | awk \u0026#39;{print $1}\u0026#39;) write-kubeconfig-mode: \u0026#34;0600\u0026#34; tls-san: - \u0026#34;localhost\u0026#34; - \u0026#34;127.0.0.1\u0026#34; - \u0026#34;$NODE_01\u0026#34; - \u0026#34;$NODE_IP01\u0026#34; - \u0026#34;$NODE_02\u0026#34; - \u0026#34;$NODE_IP02\u0026#34; - \u0026#34;$NODE_03\u0026#34; - \u0026#34;$NODE_IP03\u0026#34; container-runtime-endpoint: unix:///var/run/crio/crio.sock cluster-cidr: 10.42.0.0/16 service-cidr: 10.43.0.0/16 # Make a etcd snapshot every 6 hours etcd-snapshot-schedule-cron: \u0026#34; */6 * * *\u0026#34; # Keep 56 etcd snapshorts (equals to 2 weeks with 6 a day) etcd-snapshot-retention: 56 EOF if everything is ready, run the service and apply it to run when the vm boots\nsystemctl enable --now k3s.service Apply Canal CNI (Calico for policy and flannel for networking)\ncurl -sSL https://raw.githubusercontent.com/projectcalico/calico/v3.30.1/manifests/canal.yaml -o $HOME/canal.yaml sed -i \u0026#39;s|10.244.0.0/16|10.42.0.0/16|g\u0026#39; $HOME/canal.yaml cp $HOME/canal.yaml /var/lib/rancher/k3s/server/manifests/canal.yaml Join another control-plane node to the cluster Get cluster token in first node in /var/lib/rancher/k3s/server/agent-token looks like this\n[root@litekube01 ~]# cat /var/lib/rancher/k3s/server/agent-token K10bb908f13304637e4547ad7a40edac1c3c25165c6812c65a36f327fb99d33aa86::server:359dbfc03ab8ef930664d10adeb45397 curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_CHANNEL=v1.32.5+k3s1 sh -s - server Same as the first node, create folder /etc/rancher/k3s then create configuration file based on the environmental variables we defined earlier but we add token and server values.\nmkdir -p /etc/rancher/k3s cat \u0026lt;\u0026lt;EOF | tee /etc/rancher/k3s/config.yaml server: https://$NODE_01:6443 token: K10bb908f13304637e4547ad7a40edac1c3c25165c6812c65a36f327fb99d33aa86::server:359dbfc03ab8ef930664d10adeb45397 node-ip: $(hostname -I | awk \u0026#39;{print $1}\u0026#39;) write-kubeconfig-mode: \u0026#34;0600\u0026#34; tls-san: - \u0026#34;localhost\u0026#34; - \u0026#34;127.0.0.1\u0026#34; - \u0026#34;$NODE_01\u0026#34; - \u0026#34;$NODE_IP01\u0026#34; - \u0026#34;$NODE_02\u0026#34; - \u0026#34;$NODE_IP02\u0026#34; - \u0026#34;$NODE_03\u0026#34; - \u0026#34;$NODE_IP03\u0026#34; container-runtime-endpoint: unix:///var/run/crio/crio.sock cluster-cidr: 10.42.0.0/16 service-cidr: 10.43.0.0/16 # Make a etcd snapshot every 6 hours etcd-snapshot-schedule-cron: \u0026#34; */6 * * *\u0026#34; # Keep 56 etcd snapshorts (equals to 2 weeks with 6 a day) etcd-snapshot-retention: 56 EOF Post Bootsraping Cluster In this step execution on all nodes\nPATH update .bashrc file in $HOME/.bashrc and /etc/skel/.bashrc with text editor then adjust like this:\n#export PATH export PATH=$PATH:/usr/local/bin Create and link or copy cluster credential file.\nmkdir -p $HOME/.kube ln -s /etc/rancher/k3s/k3s.yaml $HOME/.kube/config Create cri-tools configuration file.\ncat \u0026lt;\u0026lt;EOF \u0026gt; /etc/crictl.yaml runtime-endpoint: unix:///var/run/crio/crio.sock image-endpoint: unix:///var/run/crio/crio.sock timeout: 10 debug: false EOF Installing helm latest version\nHELM_VER=$(curl -sL https://get.helm.sh/helm-latest-version) HELM_FILE=helm-$HELM_VER-linux-amd64.tar.gz curl -sSL https://get.helm.sh/helm-$HELM_VER-linux-amd64.tar.gz -o $HELM_FILE; tar -zxf $HELM_FILE mv linux-amd64/helm /usr/local/bin Update bash completion for easy cluster administration\nfor i in kubectl crictl helm; do $i completion bash | tee /etc/bash_completion.d/$i \u0026gt; /dev/null; done source /usr/share/bash-completion/bash_completion Now you can administrating k3s cluster and debugging kubernetes nodes with crictl looks like this: Agent/Worker join node to the cluster When we need horizontal cluster scaling we can add agent/worker nodes. Download and install k3s for agent and copy cluster token in first node $NODE_01 in /var/lib/rancher/k3s/server/agent-token\ncurl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.32.5+k3s1 sh -s - agent --server https://$NODE_01:6443 \\ --token K10bb908f13304637e4547ad7a40edac1c3c25165c6812c65a36f327fb99d33aa86::server:359dbfc03ab8ef930664d10adeb45397 Referensi: https://docs.k3s.io/cli/server https://docs.k3s.io/datastore/ha-embedded https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/install-for-flannel https://blog.andreev.it/2023/10/install-kubernetes-with-cri-o-and-cilium-on-rocky-linux-9 ","permalink":"http://vikipranata.is-a.dev/posts/k3s-multi-master/","summary":"\u003ch2 id=\"preparation\"\u003e\u003cem\u003ePreparation\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eI used three VM nodes for this home lab project with 4 Cores 4GB Memory and 20GB for the containers storage with operating systems \u003cem\u003eRocky Linux 9.5\u003c/em\u003e with kubernetes \u003cem\u003ev1.32.5+k3s1\u003c/em\u003e and cri-o \u003cem\u003ev1.32\u003c/em\u003e.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e\u003cem\u003eNode Hostname\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003evCPU\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eMemory\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eStorage\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003ePrivateNet\u003c/em\u003e\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003elitekube01\u003c/td\u003e\n          \u003ctd\u003e4 Core\u003c/td\u003e\n          \u003ctd\u003e4GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.111\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003elitekube02\u003c/td\u003e\n          \u003ctd\u003e4 Core\u003c/td\u003e\n          \u003ctd\u003e4GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.112\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003elitekube03\u003c/td\u003e\n          \u003ctd\u003e4 Core\u003c/td\u003e\n          \u003ctd\u003e4GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.113\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cp\u003eAll operations use the \u003ccode\u003eroot\u003c/code\u003e user, be careful when running commands!\u003cbr\u003e\nIn this step execution on all nodes\u003c/p\u003e","title":"Lightweight Kubernetes Cluster with Multi Masters K3S and CRI-O"},{"content":"Preparation I used three VM nodes for this home lab project with 4 Cores 4GB Memory and 20GB for the containers storage with operating systems Rocky Linux 9.5 with Docker community edition v28.2.2.\nNode Hostname vCPU Memory Storage PrivateNet hmlab01swarm01 4 Core 4GB 20GB 172.16.0.111 hmlab01swarm02 4 Core 4GB 20GB 172.16.0.112 hmlab01swarm03 4 Core 4GB 20GB 172.16.0.113 All operations use the root user, be careful when running commands!\nIn this step execution on all nodes\nOptional: Set SElinux to permissive mode for dev environments\nsetenforce 0 sed -i \u0026#39;s/^SELINUX=enforcing$/SELINUX=permissive/\u0026#39; /etc/selinux/config Set static hostname Set Up environment\nnode_01=hmlab01swarm01 node_02=hmlab01swarm02 node_03=hmlab01swarm03 node_IP01=172.16.0.111 node_IP02=172.16.0.112 node_IP03=172.16.0.113 then add static hostname in /etc/hosts for all nodes.\ncat \u0026lt;\u0026lt;EOF \u0026gt; /etc/hosts $NODE_IP01 $NODE_01 $NODE_IP02 $NODE_02 $NODE_IP03 $NODE_03 EOF Add Repository and Install Docker dnf install -y dnf-plugins-core dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin Set\ncat \u0026lt;\u0026lt;EOF | sudo tee /etc/docker/daemon.json { \u0026#34;exec-opts\u0026#34;: [\u0026#34;native.cgroupdriver=systemd\u0026#34;], \u0026#34;log-driver\u0026#34;: \u0026#34;json-file\u0026#34;, \u0026#34;log-opts\u0026#34;: { \u0026#34;max-size\u0026#34;: \u0026#34;100m\u0026#34; }, \u0026#34;storage-driver\u0026#34;: \u0026#34;overlay2\u0026#34; } EOF systemctl restart docker.service systemctl enable docker.service Bootsraping cluster First execute only in first manager node\ndocker swarm init --advertise-addr 172.16.0.111 --force-new-cluster To add other manager ndoe exec this command that we have before with docker swarm join-token manager in first manager node.\ndocker swarm join --token SWMTKN-1-04x40p3y8w72nv7a3merg1ee4ya4t6r3ra379gv7hv79udt0ho-8d18zrgpy5elj3f69i7rvz8tf 172.16.0.111:2377 References https://docs.rockylinux.org/gemstones/containers/docker https://docs.docker.com/reference/cli/docker/swarm https://dockerlabs.collabnix.com/intermediate/Implementing_High_Availability_with_Docker_Swarm.html ","permalink":"http://vikipranata.is-a.dev/posts/setup-docker-swarm/","summary":"\u003ch2 id=\"preparation\"\u003e\u003cem\u003ePreparation\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eI used three VM nodes for this home lab project with 4 Cores 4GB Memory and 20GB for the containers storage with operating systems \u003cem\u003eRocky Linux 9.5\u003c/em\u003e with Docker community edition \u003cem\u003ev28.2.2\u003c/em\u003e.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e\u003cem\u003eNode Hostname\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003evCPU\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eMemory\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eStorage\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003ePrivateNet\u003c/em\u003e\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ehmlab01swarm01\u003c/td\u003e\n          \u003ctd\u003e4 Core\u003c/td\u003e\n          \u003ctd\u003e4GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.111\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ehmlab01swarm02\u003c/td\u003e\n          \u003ctd\u003e4 Core\u003c/td\u003e\n          \u003ctd\u003e4GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.112\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ehmlab01swarm03\u003c/td\u003e\n          \u003ctd\u003e4 Core\u003c/td\u003e\n          \u003ctd\u003e4GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e172.16.0.113\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cp\u003eAll operations use the \u003ccode\u003eroot\u003c/code\u003e user, be careful when running commands!\u003cbr\u003e\nIn this step execution on all nodes\u003c/p\u003e","title":"Build Docker Swarm Cluster with Rocky Linux 9.5"},{"content":"Finally, VyOS Stream has been released in the first quarter of this year!\nUnlike the rolling release, the stream branch will only receive features we can confidently include in future LTS releases.\nI used it to do some testing on my home lab, so here it is.\nPreparation I used two VM nodes for this home lab project with 1 Cores 1GB Memory and 10GB for the root disk with operating systems VyOS 1.5 Stream 2025 Q1.\nNode Hostname Node Role vCPU Memory RootDisk privateNet btnlab01rtr01 Master Router 1 Core 1 GB 10 GB 10.78.78.251 btnlab01rtr02 Secondary Router 1 Core 1 GB 10 GB 10.78.78.251 Interface and network mapping:\neth0 198.51.100.253/24 gateway 198.51.100.254 for virtual IP public network eth1 10.78.78.0/24 gateway 10.78.78.254 for private network Installation Unfortunately, VyOS Stream only provides generic ISO images; there is no additional flavor. So the only way right now is for you to boot the ISO to a CD-ROM on the virtual machine. Once done, log in to live ISO with default user vyos and password vyos and see the instructions seen in the image below\nthen reboot and remove ISO from CD-ROM.\nInitialize Setup Execute in all nodes.\nEnter configuration mode with configure command\nIn this section we configure common or global configuration for each router.\nset service ssh port 22 set system name-server 1.1.1.1 set system name-server 1.0.0.1 set system time-zone Asia/Jakarta set system option reboot-on-panic set system option time-format 24-hour set system option performance throughput del service ntp set service ntp server time.cloudflare.com Additional, add some banners before and after login.\nset system login banner post-login \u0026#34;=============================================================================================================\\nATTENTION!! UNAUTHORIZED ACCESS TO THIS DEVICE IS PROHIBITED.\\nYou must have explicit, authorized permission to access or configure this device.\\nUnauthorized attempts and actions to access or use this system may result in civil and/or criminal penalties.\\nAll activities performed on this device are logged and monitored.\\n=============================================================================================================\u0026#34; set system login banner pre-login \u0026#34;=============================================================================================================\\nATTENTION!! UNAUTHORIZED ACCESS TO THIS DEVICE IS PROHIBITED.\\nYou must have explicit, authorized permission to access or configure this device.\\nUnauthorized attempts and actions to access or use this system may result in civil and/or criminal penalties.\\nAll activities performed on this device are logged and monitored.\\n=============================================================================================================\u0026#34; Additional, replace default vyos user and crate user vikiadm and add ssh public key.\nset system login user vikiadm authentication encrypted-password \u0026#39;$6$rounds=4096$Fp08IS0HCcUvA0ln$3h.4UN9EXy4Pevt8McdNqajmpDxax2cvg6k01lnxCdZgI8d4AS833fSdnHH70dDFf2zkhmJT3fnGc0ZQLV1Eh1\u0026#39; set system login user vikiadm authentication public-keys admin@tworty.com type ssh-ed25519 set system login user vikiadm authentication public-keys admin@tworty.com key AAAAC3NzaC1lZDI1NTE5AAAAIHgl+ZYCgd4eq+aMMO4uD9sjFv+tjFKFfAk+cvaYxzcq Configure common firewall and nat rule, for now we allow all protocol incoming packet and traffic in interface eth1.\nset firewall ipv4 input filter default-action drop set firewall ipv4 input filter rule 10 action accept set firewall ipv4 input filter rule 10 protocol icmp set firewall ipv4 input filter rule 10 state established set firewall ipv4 input filter rule 10 state related set firewall ipv4 input filter rule 10 state new set firewall ipv4 input filter rule 99 action accept set firewall ipv4 input filter rule 99 protocol all set firewall ipv4 input filter rule 99 inbound-interface name eth1 set firewall ipv4 input filter rule 99 source group network-group privateNet set firewall ipv4 input filter rule 99 description allow-privateNet set firewall ipv4 input filter rule 100 action accept set firewall ipv4 input filter rule 100 protocol tcp set firewall ipv4 input filter rule 100 destination port 22 set firewall ipv4 input filter rule 100 description allow-ssh set nat source rule 100 description privateNet set nat source rule 100 source address 10.78.78.0/24 set nat source rule 100 outbound-interface name eth0 set nat source rule 100 translation address masquerade Don\u0026rsquo;t forget to commit the current set of changes and save configuration\nHigh Availibility Transition Script Create transition script for vrrp service to handle public ip and default gateway for private network.\ntransition-script for become master\ncat \u0026lt;\u0026lt;EOF | tee /config/scripts/vrrp-master.sh #!/bin/vbash if [ \u0026#34;$(id -g -n)\u0026#34; != \u0026#39;vyattacfg\u0026#39; ] ; then exec sg vyattacfg -c \u0026#34;/bin/vbash \\$(readlink -f \\$0) \\$@\u0026#34; fi source /opt/vyatta/etc/functions/script-template configure del protocols static route 0.0.0.0/0 next-hop 10.78.78.254 set protocols static route 0.0.0.0/0 next-hop 198.51.100.254 commit save exit EOF transition-script for become backup and fault\ncat \u0026lt;\u0026lt;EOF | tee /config/scripts/vrrp-backup.sh #!/bin/vbash if [ \u0026#34;$(id -g -n)\u0026#34; != \u0026#39;vyattacfg\u0026#39; ] ; then exec sg vyattacfg -c \u0026#34;/bin/vbash \\$(readlink -f \\$0) \\$@\u0026#34; fi source /opt/vyatta/etc/functions/script-template configure del protocols static route 0.0.0.0/0 next-hop 198.51.100.254 set protocols static route 0.0.0.0/0 next-hop 10.78.78.254 commit save exit EOF Master Router Execute in master only.\nEnter configuration mode with configure command\nSetup hostname and interface and IP address.\nset system host-name btnlab01vrtr01 set interface ethernet eth0 description publicNet set interface ethernet eth1 description privateNet set interface ethernet eth1 address 10.78.78.251/24 Setup config-sync for Secondary/econdary router, at this point we only need a few configurations that need to be synchronized at the section config.\nset service config-sync mode load set service config-sync secondary address 10.78.78.252 set service config-sync secondary port 22940 set service config-sync secondary key config-sync-key set service config-sync section firewall set service config-sync section nat set service config-sync section pki set service config-sync section policy set service config-sync section vpn set service config-sync section service ntp set service config-sync section qos interfaces set service config-sync section qos policy set service config-sync section interfaces wireguard set service config-sync section system time-zone set service config-sync section system option set service config-sync section system static-host-mapping Setup high availibility vrrp to secondary router is basically like the keepalived configuration. Higher priority value increases router’s precedence in the master elections.\nset high-availability vrrp group defaultHA interface eth1 set high-availability vrrp group defaultHA hello-source-address 10.78.78.251 set high-availability vrrp group defaultHA peer-address 10.78.78.252 set high-availability vrrp group defaultHA address 10.78.78.254/24 interface eth1 set high-availability vrrp group defaultHA address 103.150.80.130/28 interface eth0 set high-availability vrrp group defaultHA no-preempt set high-availability vrrp group defaultHA priority 101 set high-availability vrrp group defaultHA track interface eth1 set high-availability vrrp group defaultHA vrid 96 set high-availability vrrp sync-group sync member defaultHA set high-availability vrrp sync-group sync health-check ping 10.78.78.252 set high-availability vrrp sync-group sync health-check interval 10 set high-availability vrrp sync-group sync health-check failure-count 3 set high-availability vrrp sync-group sync transition-script master \u0026#39;/config/scripts/vrrp-master.sh defaultHA\u0026#39; set high-availability vrrp sync-group sync transition-script fault \u0026#39;/config/scripts/vrrp-backup.sh defaultHA\u0026#39; set high-availability vrrp sync-group sync transition-script backup \u0026#39;/config/scripts/vrrp-backup.sh defaultHA\u0026#39; Configure conntrack-sync and enable helpers\nset service conntrack-sync accept-protocol tcp set service conntrack-sync accept-protocol udp set service conntrack-sync accept-protocol icmp set service conntrack-sync event-listen-queue-size 8 set service conntrack-sync failover-mechanism vrrp sync-group sync set service conntrack-sync interface eth1 set service conntrack-sync mcast-group 224.0.0.50 set service conntrack-sync sync-queue-size 8 used for stateful failover without conntrack synchronization, the backup firewall will see the active connection as “new” on failover, potentially disconnecting the session (e.g. VPN, banking).\nDon\u0026rsquo;t forget to commit the current set of changes and save configuration\nSecondary Router Execute in master only.\nEnter configuration mode with configure command\nSetup hostname and interface and IP address.\nset system host-name btnlab01vrtr02 set interface ethernet eth0 description publicNet set interface ethernet eth1 description privateNet set interface ethernet eth1 address 10.78.78.252/24 Setup https service to provide config-sync via api\nset service https port 22940 set service https listen-address 10.78.78.251 set service https allow-client address 10.78.78.252 Setup high availibility vrrp to master router is basically like the keepalived configuration. Higher priority value increases router’s precedence in the master elections.\nset high-availability vrrp group defaultHA interface eth1 set high-availability vrrp group defaultHA hello-source-address 10.78.78.252 set high-availability vrrp group defaultHA peer-address 10.78.78.251 set high-availability vrrp group defaultHA address 10.78.78.254/24 interface eth1 set high-availability vrrp group defaultHA address 103.150.80.130/28 interface eth0 set high-availability vrrp group defaultHA no-preempt set high-availability vrrp group defaultHA priority 100 set high-availability vrrp group defaultHA track interface eth1 set high-availability vrrp group defaultHA vrid 96 set high-availability vrrp sync-group sync member defaultHA set high-availability vrrp sync-group sync health-check ping 10.78.78.251 set high-availability vrrp sync-group sync health-check interval 10 set high-availability vrrp sync-group sync health-check failure-count 3 set high-availability vrrp sync-group sync transition-script master \u0026#39;/config/scripts/vrrp-master.sh defaultHA\u0026#39; set high-availability vrrp sync-group sync transition-script fault \u0026#39;/config/scripts/vrrp-backup.sh defaultHA\u0026#39; set high-availability vrrp sync-group sync transition-script backup \u0026#39;/config/scripts/vrrp-backup.sh defaultHA\u0026#39; Don\u0026rsquo;t forget to commit the current set of changes and save configuration\nOperational Command if you run this command in configuration mode please add run first before command execute.\nTo see firewall rules exec show firewall and show firewall statistics To see nat masquerade rules exec show nat source rules and show nat source statistics To see high availibility status show vrrp and show vrrp statistics and if you want to test the Virtual IP transition, you can do restart vrrp command.\nIn my experience testing this VyOS Stream release is very comfortable and better performance than rolling releases that must be tested periodically, I think this is enough to do PoC on the staging environment before implementing it directly in the production environment. In the future, I might use it as a reverse proxy server and vpn server with a combination of prepared automation systems such as API, Ansible, Terraform, Cloud-init, etc.\nReferences https://docs.vyos.io/en/latest/ https://docs.vyos.io/en/latest/configuration/highavailability/ https://docs.vyos.io/en/latest/automation/command-scripting.html https://docs.vyos.io/en/latest/configuration/service/config-sync.html https://vyos.net/get/stream/#1.5-2025-Q1 ","permalink":"http://vikipranata.is-a.dev/posts/vyos-router-setup-ha/","summary":"\u003cp\u003eFinally, VyOS Stream has been released in the first quarter of this year!\u003cbr\u003e\nUnlike the rolling release, the stream branch will only receive features we can confidently include in future LTS releases.\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"vyos-stream\" loading=\"lazy\" src=\"/posts/vyos-router-setup-ha/uploads/vyos-stream.webp\"\u003e\u003c/p\u003e\n\u003cp\u003eI used it to do some testing on my home lab, so here it is.\u003c/p\u003e\n\u003ch2 id=\"preparation\"\u003e\u003cem\u003ePreparation\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eI used two VM nodes for this home lab project with 1 Cores 1GB Memory and 10GB for the root disk with operating systems VyOS 1.5 Stream 2025 Q1.\u003c/p\u003e","title":"High Availabililty VyOS Stream with Config Sync"},{"content":"After we finish building the openstack cluster with kolla ansible in the previous post, to operate OpenStack with CLI there are several steps as follows:\nAccessing the Cluster Using the OpenStack RC File The RC file contains a collection of variables used to access OpenStack with specific users or projects.\nsource ~/admin-openrc.sh Creating a Project openstack project create --enable --description \u0026#34;project for kubernetes\u0026#34; kubernetes openstack project list --long --fit-width Creating a User openstack user create --project admin --email viki@homelab.is-a.dev --password p@ssw0rd viki openstack user create --project kubernetes --email k8s@homelab.is-a.dev --password-prompt k8s openstack user set viki --project kubernetes Verification\nfor i in {viki,k8s}; do openstack user show $i; done Project Role openstack role add --user viki --project admin admin openstack role add --user viki --project kubernetes member openstack role add --user k8s --project kubernetes admin Verification\nfor i in {viki,k8s}; do openstack role assignment list --user $i --names; done Creating a Project Quota openstack quota set --core 24 --ram 20480 --instances 10 --volumes 10 --floating-ips 6 --secgroups 2 kubernetes Creating Images Download the cloud image file first:\nwget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img curl -LO https://download.cirros-cloud.net/0.6.1/cirros-0.6.1-x86_64-disk.img Then import the downloaded image into OpenStack using the following command:\nopenstack image create --public --disk-format qcow2 --file cirros-0.6.1-x86_64-disk.img cirros-test openstack image create --public --disk-format qcow2 --file focal-server-cloudimg-amd64.img ubuntu20-focal openstack image list Creating a Flavor A flavor defines the specifications of an instance. For example, creating a flavor with 1 VCPU, 1GB RAM, and 10GB Disk can be done with the following command:\nopenstack flavor create --project kubernetes --private --vcpu 2 --ram 2048 --disk 2 n1-kubemaster openstack flavor create --project kubernetes --private --vcpu 4 --ram 4096 --disk 4 n2-kubeworker openstack flavor list --all Creating an SSH Keypair A keypair in this context refers to an SSH keypair that we usually create with the ssh-keygen command, which generates a public key and private key file. This helps cloud-init to insert the public key into the instance. To create a keypair, use the following command:\nopenstack keypair create --public-key ~/.ssh/id_rsa.pub controller-key openstack keypair create --public-key ~/.ssh/k8s.pub k8s-key Creating a Security Group A security group acts as a firewall at the instance level. For example, we can set up incoming ingress connections on SSH (TCP port 22), web services (TCP ports 80, 443), and ICMP protocol with the following commands:\nCreating kubesecgroup security group\nopenstack security group create kubesecgroup openstack security group rule create --ingress --protocol icmp openstack security group rule create --ingress --protocol tcp --dst-port 22 --description ssh openstack security group rule create --ingress --protocol tcp --dst-port 80 --description http openstack security group rule create --ingress --protocol tcp --dst-port 443 --description https openstack security group rule create --ingress --protocol tcp --dst-port 6443 --description api-service openstack security group rule create --ingress --protocol tcp --dst-port 8443 --description ha-api-service openstack security group show kubesecgroup Creating app-secgroup security group\nopenstack security group create app-secgroup openstack security group rule create --ingress --protocol tcp --dst-port 30080 --description http-ingress openstack security group rule create --ingress --protocol tcp --dst-port 30443 --description https-ingress Creating an External Network An external network is used for communication out of the instance via an external provider. To create one, we need to know the provider physical network first with the following command:\nsudo cat /etc/kolla/neutron-server/ml2_conf.ini | grep flat_network | awk \u0026#39;{print $3}\u0026#39; Then, define the network and subnet with these commands:\nopenstack network create --project admin --external --provider-network-type flat \\ --provider-physical-network physnet1 ext-net openstack subnet create --network ext-net \\ --subnet-range 172.16.1.0/24 --gateway 172.16.1.1 --dns-nameserver 172.16.1.1 \\ --allocation-pool start=172.16.1.242,end=172.16.1.254 --no-dhcp ext-subnet openstack network list --long --fit-width openstack subnet list --long --fit-width Creating an Internal Network An internal network is used for communication between instances over an internal network. To create one, define the network and subnet with the following commands:\nopenstack network create --internal kubenet openstack subnet create --network kubenet --subnet-range 10.1.0.0/24 --gateway 10.1.0.1 kubesubnet openstack network list --long --fit-width openstack subnet list --long --fit-width Creating a router In OpenStack, a router is used to connect internal traffic to external traffic, whether it\u0026rsquo;s incoming (ingress) or outgoing (egress). To create a router, use the following commands:\nopenstack router create kuberouter openstack router set --external-gateway ext-net kuberouter openstack router add subnet kuberouter kubesubnet openstack router show kuberouter Creating an Instance Once the above steps are in place, you can create an instance with the following commands:\nopenstack server create --flavor n1-kubemaster --network kubenet \\ --key-name k8s-key --image ubuntu20-focal --security-group kubesecgroup k8s-master01 openstack server create --flavor n2-kubeworker --network kubenet \\ --key-name k8s-key --image ubuntu20-focal --security-group kubesecgroup k8s-worker01 Creating a Floating IP A floating IP functions as a NAT applied to the internal IP of an instance. To create floating IPs, use the following commands:\nopenstack floating ip create --floating-ip 172.16.1.244 ext-net openstack floating ip create --floating-ip 172.16.1.245 ext-net openstack floating ip list openstack server add floating ip k8s-master01 172.16.1.244 openstack server add floating ip k8s-worker01 172.16.1.245 openstack server list Creating a Volume A volume is used as persistent storage for an instance in the form of block storage. To create a volume and attach it to an instance, use the following commands:\nopenstack volume create --size 1 mvolume openstack server add volume k8s-master01 mvolume Expanding a Volume openstack server remove volume k8s-master01 mvolume openstack volume set --size 2 mvolume openstack volume list openstack server add volume k8s-master01 mvolume openstack server show k8s-master01 Instance with Persistent Storage To create persistent storage on the root partition when an instance is created, use the following command:\nopenstack volume create --size 10 --image ubuntu20-focal web-server-ubuntu02 openstack server create --flavor c1-standard-01 \\ --key-name controllerkey \\ --security-group secg-basic-web \\ --network int-net01 \\ --volume web-server-ubuntu02 --wait \\ web-server-ubuntu02 Upgrading an Instance with a New Flavor Over time, the specifications of an instance may need to be upgraded to meet new resource requirements. To apply this upgrade, follow these commands:\nopenstack flavor create --vcpus 2 --ram 2048 --disk 15 --public c2-standard-01 openstack server resize --flavor c2-standard-01 web-server-ubuntu01 openstack server list | grep web-server-ubuntu01 openstack server resize confirm web-server-ubuntu01 References https://docs.openstack.org/python-openstackclient/latest/cli https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/project-v2.html#project-create https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/user-v2.html#user-create https://docs.openstack.org/keystone/rocky/admin/cli-manage-projects-users-and-roles.html ","permalink":"http://vikipranata.is-a.dev/posts/openstack-operational-cli/","summary":"\u003cp\u003eAfter we finish building the openstack cluster with kolla ansible in the \u003ca href=\"/posts/openstack-kolla-ansible-2024.2/\"\u003eprevious post\u003c/a\u003e, to operate OpenStack with CLI there are several steps as follows:\u003c/p\u003e\n\u003ch2 id=\"accessing-the-cluster\"\u003eAccessing the Cluster\u003c/h2\u003e\n\u003ch3 id=\"using-the-openstack-rc-file\"\u003eUsing the OpenStack RC File\u003c/h3\u003e\n\u003cp\u003eThe RC file contains a collection of variables used to access OpenStack with specific users or projects.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003esource\u003c/span\u003e ~/admin-openrc.sh\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"creating-a-project\"\u003eCreating a Project\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eopenstack project create --enable --description \u003cspan class=\"s2\"\u003e\u0026#34;project for kubernetes\u0026#34;\u003c/span\u003e kubernetes\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eopenstack project list --long --fit-width\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"creating-a-user\"\u003eCreating a User\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eopenstack user create --project admin --email viki@homelab.is-a.dev --password p@ssw0rd viki\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eopenstack user create --project kubernetes --email k8s@homelab.is-a.dev --password-prompt k8s\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eopenstack user \u003cspan class=\"nb\"\u003eset\u003c/span\u003e viki --project kubernetes\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eVerification\u003c/p\u003e","title":"Operating OpenStack via CLI"},{"content":"Preparation I used five VM nodes for this home lab project with 16 Cores 16GB Memory and 40GB for the root disk with operating systems Ubuntu 22.04.5 LTS.\nNode Hostname Node Role vCPU Memory RootDisk ManagementNet StorageNet btnlab01adm01 Ansible 2 Core 2GB 20GB 10.78.78.199 - btnlab01con01 Controller 16 Core 16GB 40GB 10.78.78.201 10.79.79.201 btnlab01con02 Controller 16 Core 16GB 40GB 10.78.78.202 10.79.79.202 btnlab01con03 Controller 16 Core 16GB 40GB 10.78.78.203 10.79.79.203 btnlab01hpv01 Hypervisor 32 Core 32GB 40GB 10.78.78.204 10.79.79.204 btnlab01hpv02 Hypervisor 32 Core 32GB 40GB 10.78.78.205 10.79.79.205 Interface mapping:\neth0 for Public Network eth1 for Management Network eth2 for Storage Network with Jumbo frames eth4 for Tunnel/Self Service/Guest Network with Jumbo frames We can review network configuration in sample-netplan.yaml\nInstalling NFS Server for Glance Service in btnlab01adm01 node\nsudo apt update sudo apt install nfs-kernel-server sudo mkdir -p /data sudo chown nobody:nogroup /data echo \u0026#34;/data *(rw,sync,no_subtree_check)\u0026#34; \u0026gt;\u0026gt; /etc/exports sudo systemctl restart nfs-kernel-server sudo exportfs -v Execute in all nodes\nMapping static hostname in /etc/hosts\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/hosts 10.78.78.199 btnlab01adm01 btnlab01adm01.homelab.is-a.dev 10.78.78.201 btnlab01con01 btnlab01con01.homelab.is-a.dev 10.78.78.202 btnlab01con02 btnlab01con02.homelab.is-a.dev 10.78.78.203 btnlab01con02 btnlab01con03.homelab.is-a.dev 10.78.78.204 btnlab01hpv01 btnlab01hpv01.homelab.is-a.dev 10.78.78.205 btnlab01hpv02 btnlab01hpv02.homelab.is-a.dev 10.78.78.200 os-int 10.78.78.100 os-ext EOF Install packages and dependencies\nsudo apt install -y git gcc libssl-dev libffi-dev \\ python3-venv python3-dev python3-selinux python3-setuptools \\ python3-pip python3-docker nfs-common Mount nfs volume for all node\nsudo mkdir /data cat \u0026lt;\u0026lt;EOF | sudo tee -a /etc/fstab 10.78.78.199:/data /data nfs defaults,_netdev 0 0 EOF sudo mount /data Create virtual environment for kolla-ansible\npython3 -m venv kolla-venv cd kolla-venv source bin/activate pwd Update pip and install ansible\npip install -U pip pip install \u0026#39;ansible\u0026gt;=8,\u0026lt;9\u0026#39; Update ansible config in current directory\ntee $PWD/ansible.cfg \u0026lt;\u0026lt;EOF [defaults] host_key_checking=False pipelining=True forks=100 log_path = $PWD/ansible.log EOF Installing kolla-ansible stable with version 2024.1\npip install git+https://opendev.org/Openstack/kolla-ansible@stable/2024.1 kolla-ansible install-deps Create directory for kolla configuration\nsudo mkdir -p /etc/kolla /etc/kolla/config/nova \\ /etc/kolla/config/cinder/{cinder-volume,cinder-backup} sudo chown $USER:$USER /etc/kolla Copy ceph keyring into nova and cinder config directory that we created earlier in the Ceph Reef Deployment post.\nsudo mkdir -p /etc/kolla /etc/kolla/config/nova \\ /etc/kolla/config/cinder/{cinder-volume,cinder-backup} cp -r ceph.conf \\ /etc/kolla/config/{nova,cinder}/ceph.conf cp -r ceph.client.cinder.keyring \\ /etc/kolla/config/nova/ceph.client.cinder.keyring cp -r ceph.client.cinder.keyring \\ /etc/kolla/config/cincer/cinder-volume/ceph.client.cinder.keyring cp -r ceph.client.cinder.keyring \\ /etc/kolla/config/cincer/cinder-backup/ceph.client.cinder-backup.keyring sudo chown -R $USER:$USER /etc/kolla Then copy kolla configuration to /etc/kolla and copy multinode file into current directory\ncp -r share/kolla-ansible/etc_examples/kolla/* /etc/kolla cp -r share/kolla-ansible/ansible/inventory/multinode . Kolla Configuration Adjust inventory multinode file, sometimes like this\n[control] btnlab01con01 btnlab01con02 btnlab01con03 [network] btnlab01con01 btnlab01con02 btnlab01con03 [compute] btnlab01hpv01 btnlab01hpv02 [monitoring] btnlab01con01 btnlab01con02 btnlab01con03 Then validate with ansible ping for all node in inventory multinode\nansible -i multinode all -m ping then adjust /etc/kolla/globals.yml like this\nworkaround_ansible_issue_8743: yes kolla_base_distro: \u0026#34;ubuntu\u0026#34; kolla_install_type: \u0026#34;source\u0026#34; Openstack_release: \u0026#34;2024.1\u0026#34; kolla_internal_vip_address: \u0026#34;10.78.78.200\u0026#34; kolla_internal_fqdn: \u0026#34;os-int\u0026#34; kolla_external_vip_address: \u0026#34;10.78.78.100\u0026#34; kolla_external_fqdn: \u0026#34;os-ext\u0026#34; network_interface: \u0026#34;eth1\u0026#34; tunnel_interface: \u0026#34;eth3\u0026#34; neutron_external_interface: \u0026#34;eth0\u0026#34; neutron_plugin_agent: \u0026#34;ovn\u0026#34; kolla_enable_tls_internal: \u0026#34;yes\u0026#34; kolla_enable_tls_external: \u0026#34;yes\u0026#34; kolla_copy_ca_into_containers: \u0026#34;yes\u0026#34; Openstack_cacert: \u0026#34;/etc/ssl/certs/ca-certificates.crt\u0026#34; kolla_enable_tls_backend: \u0026#34;yes\u0026#34; Openstack_region_name: \u0026#34;Staging\u0026#34; enable_Openstack_core: \u0026#34;yes\u0026#34; enable_haproxy: \u0026#34;yes\u0026#34; enable_mariadb: \u0026#34;yes\u0026#34; enable_memcached: \u0026#34;yes\u0026#34; enable_cinder: \u0026#34;yes\u0026#34; enable_cinder_backup: \u0026#34;yes\u0026#34; enable_mariabackup: \u0026#34;yes\u0026#34; ceph_cinder_user: \u0026#34;cinder\u0026#34; ceph_cinder_keyring: \u0026#34;client.{{ ceph_cinder_user }}.keyring\u0026#34; ceph_cinder_pool_name: \u0026#34;cinder\u0026#34; ceph_cinder_backup_user: \u0026#34;cinder-backup\u0026#34; ceph_cinder_backup_keyring: \u0026#34;client.{{ ceph_cinder_backup_user }}.keyring\u0026#34; ceph_cinder_backup_pool_name: \u0026#34;cinder-backup\u0026#34; ceph_nova_keyring: \u0026#34;{{ ceph_cinder_keyring }}\u0026#34; ceph_nova_user: \u0026#34;{{ ceph_cinder_user }}\u0026#34; ceph_nova_pool_name: \u0026#34;nova\u0026#34; glance_backend_file: \u0026#34;yes\u0026#34; glance_file_datadir_volume: \u0026#34;/data/glance\u0026#34; cinder_backend_ceph: \u0026#34;yes\u0026#34; nova_backend_ceph: \u0026#34;yes\u0026#34; nova_compute_virt_type: \u0026#34;kvm\u0026#34; Generate password for all Openstack service and dependencies\nkolla-genpwd This command will create a new file in /etc/kolla/passwords.yml\nGenerate Openstack certificate with kolla-ansible then append to system host ca-certificate\nkolla-ansible -i multinode certificates cat /etc/kolla/certificates/ca/root.crt | sudo tee -a /etc/ssl/certs/ca-certificates.crt Deploy Openstack Bootstrap servers with kolla deploy dependencies\nkolla-ansible -i ./multinode bootstrap-servers Pull all images for containers (only pulls, no running container changes)\nkolla-ansible -v -i multinode pull Do pre-deployment checks for hosts before deploy\nkolla-ansible -i ./multinode prechecks Deploy and start all kolla containers\nkolla-ansible -i multinode deploy -v Validate generated configuration files of enabled services\nkolla-ansible -i multinode validate-config Post Deploy Openstack Do post deploy\nkolla-ansible -i multinode post-deploy If all goes well there should be no count failed like this Copy admin-openrc.sh to home directory and add Openstack CA Certificate\ncp /etc/kolla/admin-openrc.sh $HOME/admin-openrc.sh echo \u0026#34;export OS_CACERT=/etc/ssl/certs/ca-certificates.crt\u0026#34; \u0026gt;\u0026gt; $HOME/admin-openrc.sh Install Openstack cli client and use admin-openrc.sh credentials for access Openstack cluster\nsudo apt install -y python3-Openstackclient source $HOME/admin-openrc.sh Do some checking commands like\nOpenstack host list Openstack hypervisor list Openstack endpoint list Openstack service list Openstack network agent list Now we can access horizon dashboard via browser https://os-ext.homelab.is-a.dev with credentials we can get from admin-openrc.sh file. Next step we can learn some basic Openstack CLI in the Operational Openstack with Command Line post.\nReferences https://docs.openstack.org/kolla-ansible/2024.2 https://docs.openstack.org/kolla-ansible/2024.2/user/quickstart-development.html https://docs.openstack.org/kolla-ansible/2024.2/user/multinode.html ","permalink":"http://vikipranata.is-a.dev/posts/openstack-kolla-ansible-2024.2/","summary":"\u003ch2 id=\"preparation\"\u003e\u003cem\u003ePreparation\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eI used five VM nodes for this home lab project with 16 Cores 16GB Memory and 40GB for the root disk with operating systems \u003cem\u003eUbuntu 22.04.5 LTS\u003c/em\u003e.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e\u003cem\u003eNode Hostname\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eNode Role\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003evCPU\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eMemory\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eRootDisk\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eManagementNet\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eStorageNet\u003c/em\u003e\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01adm01\u003c/td\u003e\n          \u003ctd\u003eAnsible\u003c/td\u003e\n          \u003ctd\u003e2 Core\u003c/td\u003e\n          \u003ctd\u003e2GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.199\u003c/td\u003e\n          \u003ctd\u003e-\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01con01\u003c/td\u003e\n          \u003ctd\u003eController\u003c/td\u003e\n          \u003ctd\u003e16 Core\u003c/td\u003e\n          \u003ctd\u003e16GB\u003c/td\u003e\n          \u003ctd\u003e40GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.201\u003c/td\u003e\n          \u003ctd\u003e10.79.79.201\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01con02\u003c/td\u003e\n          \u003ctd\u003eController\u003c/td\u003e\n          \u003ctd\u003e16 Core\u003c/td\u003e\n          \u003ctd\u003e16GB\u003c/td\u003e\n          \u003ctd\u003e40GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.202\u003c/td\u003e\n          \u003ctd\u003e10.79.79.202\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01con03\u003c/td\u003e\n          \u003ctd\u003eController\u003c/td\u003e\n          \u003ctd\u003e16 Core\u003c/td\u003e\n          \u003ctd\u003e16GB\u003c/td\u003e\n          \u003ctd\u003e40GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.203\u003c/td\u003e\n          \u003ctd\u003e10.79.79.203\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01hpv01\u003c/td\u003e\n          \u003ctd\u003eHypervisor\u003c/td\u003e\n          \u003ctd\u003e32 Core\u003c/td\u003e\n          \u003ctd\u003e32GB\u003c/td\u003e\n          \u003ctd\u003e40GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.204\u003c/td\u003e\n          \u003ctd\u003e10.79.79.204\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01hpv02\u003c/td\u003e\n          \u003ctd\u003eHypervisor\u003c/td\u003e\n          \u003ctd\u003e32 Core\u003c/td\u003e\n          \u003ctd\u003e32GB\u003c/td\u003e\n          \u003ctd\u003e40GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.205\u003c/td\u003e\n          \u003ctd\u003e10.79.79.205\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eInterface mapping:\u003c/p\u003e","title":"Openstack 2024.2 with Kolla Ansible"},{"content":"Preparation I used three VM nodes for this home lab project with 8 Cores 8GB Memory and 20GB for the root disk with Jumbo Frame in ClusterNet interface with operating systems Rocky Linux 8.10.\nNode Hostname vCPU Memory RootDisk PublicNet ClusterNet btnlab01ceph01 8 Core 8GB 20GB 10.78.78.221 10.79.79.221 btnlab01ceph02 8 Core 8GB 20GB 10.78.78.222 10.79.79.222 btnlab01ceph03 8 Core 8GB 20GB 10.78.78.223 10.79.79.223 Then I added 4 hard drives each with a capacity of 50GB used for OSDs.\nAll operations use the root user, be careful when running commands!\nOptional: Set SElinux to permissive mode for dev environments\nsetenforce 0 sed -i \u0026#39;s/^SELINUX=enforcing$/SELINUX=permissive/\u0026#39; /etc/selinux/config Install packages and dependencies dnf install -y epel-release dnf install -y bash-completion tmux nano htop iftop iotop podman lvm2 Download cephadm binary file and add ceph reef repository CEPH_RELEASE=18.2.0 curl --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm mv cephadm /usr/bin; chmod +x /usr/bin/cephadm cephadm add-repo --release $CEPH_RELEASE Bootstrapping Cluster With cephadm Execute in first node Mapping static hostname in /etc/hosts\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/hosts 10.78.78.221 btnlab01ceph01 btnlab01ceph01.homelab.is-a.dev 10.78.78.222 btnlab01ceph02 btnlab01ceph02.homelab.is-a.dev 10.78.78.223 btnlab01ceph03 btnlab01ceph03.homelab.is-a.dev EOF cephadm install cephadm install ceph-common cephadm bootstrap --mon-ip=10.78.78.221 \\ --skip-mon-network --allow-fqdn-hostname | tee cephadm-bootstrap.log Bootstrap complete success message\nCopy ssh public key to all node\nfor node in btnlab01con0{2..3}; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$node sleep 2 done After bootstrap finished add ceph public network with following this command.\nceph config set host public_network 10.79.79.0/24 Add another node then adjust placement pgs\nceph orch host add btnlab01ceph02 10.78.78.222 ceph orch host add btnlab01ceph03 10.78.78.223 ceph orch apply mgr --placement=\u0026#34;3 btnlab01ceph01 btnlab01ceph02 btnlab01ceph03\u0026#34; ceph orch apply mon --placement=\u0026#34;btnlab01ceph01,btnlab01ceph02,btnlab01ceph03\u0026#34; Label all node for node in btnlab01ceph0{1..3}; do ceph orch host label add $node mon ceph orch host label add $node mgr ceph orch host label add $node osd done Add osd from all available disk ceph orch apply osd --all-available-devices --method raw ceph orch host ls ceph orch device ls Now We can monitor ceph cluster with ceph -s command\nCreating pool name and ceph keyring for several services Cinder Cinder Backup for pool_name in cinder cinder-backup; do ceph osd pool create $pool_name rbd pool init $pool_name done ceph auth get-or-create client.cinder \\ mon \u0026#39;allow r\u0026#39; osd \u0026#39;allow class-read object_prefix rbd_children, allow rwx pool=cinder, allow rwx pool=nova\u0026#39; \\ -o /etc/ceph/ceph.client.cinder.keyring ceph auth get-or-create client.cinder-backup \\ mon \u0026#39;allow r\u0026#39; osd \u0026#39;allow class-read object_prefix rbd_children, allow rwx pool=cinder-backup\u0026#39; \\ -o /etc/ceph/ceph.client.cinder-backup.keyring We will use this keyring for the openstack cluster that we will create in the Openstack 2024.2 with Kolla Ansible post.\nNow we can access ceph dashboard via browser https://btnlab01ceph01.homelab.is-a.dev:8443 with credentials we can get from bootsrap.\nReferences https://docs.ceph.com/en/reef/cephadm/install https://docs.ceph.com/en/reef/mgr/orchestrator https://docs.ceph.com/en/reef/rados/operations/pools https://docs.ceph.com/en/reef/rados/operations/user-management ","permalink":"http://vikipranata.is-a.dev/posts/ceph-reef-deployment/","summary":"\u003ch2 id=\"preparation\"\u003e\u003cem\u003ePreparation\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eI used three VM nodes for this home lab project with 8 Cores 8GB Memory and 20GB for the root disk with \u003ca href=\"https://ceph.io/en/news/blog/2015/ceph-loves-jumbo-frames/\"\u003eJumbo Frame\u003c/a\u003e in ClusterNet interface with operating systems \u003cem\u003eRocky Linux 8.10\u003c/em\u003e.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e\u003cem\u003eNode Hostname\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003evCPU\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eMemory\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eRootDisk\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003ePublicNet\u003c/em\u003e\u003c/th\u003e\n          \u003cth\u003e\u003cem\u003eClusterNet\u003c/em\u003e\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01ceph01\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.221\u003c/td\u003e\n          \u003ctd\u003e10.79.79.221\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01ceph02\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.222\u003c/td\u003e\n          \u003ctd\u003e10.79.79.222\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ebtnlab01ceph03\u003c/td\u003e\n          \u003ctd\u003e8 Core\u003c/td\u003e\n          \u003ctd\u003e8GB\u003c/td\u003e\n          \u003ctd\u003e20GB\u003c/td\u003e\n          \u003ctd\u003e10.78.78.223\u003c/td\u003e\n          \u003ctd\u003e10.79.79.223\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eThen I added 4 hard drives each with a capacity of 50GB used for OSDs.\u003c/p\u003e","title":"Ceph Reef Deployment"},{"content":"Installing Packages Dependencies dnf install -y keepalived Keepalived Configuration Keepalived state reference\nMASTER-MASTER (if down, back to top priority) MASTER-BACKUP (if down, back to MASTER) BACKUP-BACKUP (if down, respect to node with MASTER state) Routers with priority 101 will become MASTER and Routers with priority 100 will become BACKUP. Configure First Node cat \u0026lt;\u0026lt;EOF | tee /etc/keepalived/keepalived.conf global_defs { router_id JumpServer enable_script_security vrrp_check_unicast_src } vrrp_track_process track_openvpn { process openvpn weight 2 } vrrp_instance VIP { state MASTER interface eth1 virtual_router_id 69 priority 101 advert_int 1 nopreempt authentication { auth_type PASS auth_pass Pa\\$\\$w0rd } unicast_src_ip 10.79.80.1 unicast_peer { 10.79.80.2 } virtual_ipaddress { 103.150.80.130/28 dev eth0 10.79.80.254/24 dev eth1 } virtual_routes { 0.0.0.0/0 via 103.150.80.142 dev eth0 metric 100 } static_routes { 0.0.0.0/0 via 10.79.80.251 dev eth1 metric 101 } track_process { track_openvpn } } EOF Restart service and set to run while system boot\nsystemctl restart keepalived; systemctl enable keepalived Configure Second Node cat \u0026lt;\u0026lt;EOF | tee /etc/keepalived/keepalived.conf global_defs { router_id JumpServer enable_script_security vrrp_check_unicast_src } vrrp_track_process track_openvpn { process openvpn weight 2 } vrrp_instance VIP { state BACKUP interface eth1 virtual_router_id 69 priority 100 advert_int 1 authentication { auth_type PASS auth_pass Pa\\$\\$w0rd } unicast_src_ip 10.79.80.2 unicast_peer { 10.79.80.1 } virtual_ipaddress { 104.18.5.103/28 dev eth0 10.79.80.254/24 dev eth1 } virtual_routes { 0.0.0.0/0 via 103.150.80.142 dev eth0 metric 100 } static_routes { 0.0.0.0/0 via 10.79.80.251 dev eth1 metric 101 } track_process { track_openvpn } } EOF Restart service and set to run while system boot\nsystemctl restart keepalived; systemctl enable keepalived Custom Health Check If you want to custom script for health check change or adjust this keepalived.conf file.\nvrrp_script healthcheck { script \u0026#34;/bin/bash /etc/keepalived/healthcheck.sh\u0026#34; user root root interval 2 weight 2 } vrrp_instance VIP { ... track_script { healthcheck } ... } and create healthcheck.sh bash script like this\n#!/bin/bash TARGET_URL=\u0026#34;https://127.0.0.1:443\u0026#34; USER_AGENT=$(keepalived -v 2\u0026gt;\u0026amp;1 | awk \u0026#39;/Keepalived/ {print $1\u0026#34;/\u0026#34;$2}\u0026#39;) curl --head \\ --silent \\ --insecure \\ --max-time 1 \\ --header \u0026#34;Via: $HOSTNAME\u0026#34; \\ --header \u0026#34;User-Agent: $USER_AGENT\u0026#34; \\ --request GET \u0026#34;$TARGET_URL\u0026#34; -o /dev/null echo \u0026#34;Result code $?\u0026#34; exit $? Also when SELinux is Enforcing, add this module\nsemodule -r keepalived-health-check cat \u0026lt;\u0026lt;EOF | tee keepalived-health-check.te module keepalived-health-check 1.0; require { type shell_exec_t; type keepalived_t; type keepalived_exec_t; type unreserved_port_t; type hostname_exec_t; class file { getattr setattr execute execute_no_trans open read map }; class tcp_socket name_connect; } #============= keepalived_t ============== allow keepalived_t shell_exec_t:file setattr; allow keepalived_t unreserved_port_t:tcp_socket name_connect; allow keepalived_t keepalived_exec_t:file { execute_no_trans open }; allow keepalived_t hostname_exec_t:file { getattr execute execute_no_trans open read }; #!!!! This avc can be allowed using the boolean \u0026#39;domain_can_mmap_files\u0026#39; allow keepalived_t hostname_exec_t:file map; EOF checkmodule -M -m -o keepalived-health-check.mod keepalived-health-check.te semodule_package -o keepalived-health-check.pp -m keepalived-health-check.mod semodule -i keepalived-health-check.pp sealert -a /var/log/audit/audit.log Troubleshooting with keepalived debug\nkeepalived -nldD Reference:\nhttps://www.redhat.com/sysadmin/advanced-keepalived https://access.redhat.com/solutions/3220521 https://manpages.debian.org/testing/keepalived/keepalived.conf.5.en.html https://github.com/sandervanvugt/cka/blob/master/check_apiserver.sh ","permalink":"http://vikipranata.is-a.dev/posts/high-availability-keepalived/","summary":"\u003ch1 id=\"installing-packages-dependencies\"\u003e\u003cstrong\u003eInstalling Packages Dependencies\u003c/strong\u003e\u003c/h1\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ednf install -y keepalived\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch1 id=\"keepalived-configuration\"\u003e\u003cstrong\u003eKeepalived Configuration\u003c/strong\u003e\u003c/h1\u003e\n\u003cp\u003eKeepalived state reference\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eMASTER-MASTER (if down, back to top priority)\u003c/li\u003e\n\u003cli\u003eMASTER-BACKUP (if down, back to MASTER)\u003c/li\u003e\n\u003cli\u003eBACKUP-BACKUP (if down, respect to node with MASTER state)\u003c/li\u003e\n\u003cli\u003eRouters with priority 101 will become MASTER and Routers with priority 100 will become BACKUP.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"configure-first-node\"\u003eConfigure First Node\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ecat \u003cspan class=\"s\"\u003e\u0026lt;\u0026lt;EOF | tee /etc/keepalived/keepalived.conf\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eglobal_defs {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    router_id JumpServer\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    enable_script_security\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    vrrp_check_unicast_src\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003evrrp_track_process track_openvpn {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    process openvpn\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    weight 2\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003evrrp_instance VIP {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    state MASTER\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    interface eth1\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    virtual_router_id 69\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    priority 101\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    advert_int 1\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    nopreempt\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    authentication {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e        auth_type PASS\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e        auth_pass Pa\\$\\$w0rd\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    unicast_src_ip 10.79.80.1\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    unicast_peer {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e        10.79.80.2\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    virtual_ipaddress {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e        103.150.80.130/28 dev eth0\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e        10.79.80.254/24 dev eth1\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    virtual_routes {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e        0.0.0.0/0 via 103.150.80.142 dev eth0 metric 100\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    static_routes {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e        0.0.0.0/0 via 10.79.80.251 dev eth1 metric 101\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    track_process {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e        track_openvpn\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eEOF\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eRestart service and set to run while system boot\u003c/p\u003e","title":"High Availability with Keepalived"},{"content":"Installing package dependencies dnf install -y epel-release dnf install -y openvpn openssl easy-rsa policycoreutils-python-utils Creating and Setting Up an OpenVPN Server Certificates Set of environment variable for easy_rsa\ncat \u0026lt;\u0026lt;EOF | tee vars set_var EASYRSA_ALGO ec set_var EASYRSA_CURVE prime256v1 set_var EASYRSA_CA_EXPIRE 3650 set_var EASYRSA_CERT_EXPIRE 3650 set_var EASYRSA_CRL_DAYS 3650 EOF Then execute ./easyresa init-pki\nCreate certificate authority (CA) and sever certificate.\nSERVER_CN=jumpserver SERVER_NAME=jumpserver ./easyrsa --batch --req-cn=\u0026#34;$SERVER_CN\u0026#34; build-ca nopass ./easyrsa --batch build-server-full \u0026#34;$SERVER_NAME\u0026#34; nopass ./easyrsa gen-crl Move all the generated files to /etc/openvpn/certdirectory\nmkdir -p /etc/openvpn/cert cp pki/ca.crt \\ pki/crl.pem \\ pki/private/ca.key \\ \u0026#34;pki/issued/$SERVER_NAME.crt\u0026#34; \\ \u0026#34;pki/private/$SERVER_NAME.key\u0026#34; \\ /etc/openvpn/cert Generate tls-crypt key to improve security\nopenvpn --genkey secret /etc/openvpn/server/tls-crypt.key Generate Server Configuration The default server configuration is located /usr/share/doc/openvpn/sample/sample-config-files/server.conf\ncat \u0026lt;\u0026lt;EOF | tee /etc/openvpn/server/server.conf port 1194 proto udp dev tun user nobody group nobody persist-key persist-tun keepalive 10 120 topology subnet server 10.79.83.0 255.255.255.0 push \u0026#34;route 10.79.80.0 255.255.255.0\u0026#34; push \u0026#34;dhcp-option DNS 10.79.80.254\u0026#34; push \u0026#34;dhcp-option DOMAIN lab.tworty.id\u0026#34; client-config-dir /etc/openvpn/client/ccd ifconfig-pool-persist /etc/openvpn/client/persistent-ip.list dh none ecdh-curve prime256v1 auth SHA256 cipher AES-128-GCM data-ciphers AES-128-GCM ca /etc/openvpn/cert/ca.crt cert /etc/openvpn/cert/jumpserver.crt key /etc/openvpn/cert/jumpserver.key crl-verify /etc/openvpn/cert/crl.pem tls-crypt /etc/openvpn/cert/tls-crypt.key tls-server tls-version-min 1.2 tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256 status /var/log/openvpn/status.log log-append /var/log/openvpn/openvpn.log verb 3 EOF Insert duplicate-cn to /etc/openvpn/server/server.conf if you need a multiple vpn connection with same CN. If you are using RHEL, change the group directive from nogroup to nobody to match the distribution’s available groups\nRouting, Kernel and firewall configuration Add routing policy\nnmcli device modify eth1 +ipv4.routes \u0026#34;10.79.83.0/24 10.79.80.254\u0026#34; nmcli connection modify \u0026#39;System eth1\u0026#39; +ipv4.routes \u0026#34;10.79.83.0/24 10.79.80.254\u0026#34; Add kernel parameter for ip forwarding\ncat \u0026lt;\u0026lt;EOF | tee /etc/sysctl.d/ipv4_forward.conf net.ipv4.ip_forward = 1 EOF sysctl -p Add firewall configuration using firewalld\nfirewall-cmd --get-active-zones firewall-cmd --permanent --add-service openvpn firewall-cmd --permanent --zone=internal --add-source=10.79.80.0/22 firewall-cmd --reload or spesific port and protocol with firewall-cmd --permanent --add-port=1194/udp\nSELinux configuration semanage port -a -t openvpn_port_t -p udp 1194 if you have any issue with SElinux policy please use audit sealert -a /var/log/audit/audit.log ausearch -c \u0026#39;openvpn\u0026#39; --raw | audit2allow -M openvpn-custom Generate Client Configuration The default server configuration is located /usr/share/doc/openvpn/sample/sample-config-files/client.conf\nYOUR_IP=$(curl -sL ipinfo.io/ip) cat \u0026lt;\u0026lt;EOF | tee /etc/openvpn/client/client.ovpn client proto udp explicit-exit-notify remote $YOUR_IP 1194 dev tun resolv-retry infinite nobind persist-key persist-tun remote-cert-tls server verify-x509-name jumpserver name auth SHA256 auth-nocache cipher AES-128-GCM tls-client tls-version-min 1.2 tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256 verb 3 EOF Generate Client Certificate CLIENT_NAME=jumpserver-client ./easyrsa --batch build-client-full \u0026#34;$CLIENT_NAME\u0026#34; nopass Insert cerfificate authority file\n{ echo \u0026#34;\u0026lt;ca\u0026gt;\u0026#34; cat /etc/openvpn/cert/ca.crt echo \u0026#34;\u0026lt;/ca\u0026gt;\u0026#34; } | tee -a /etc/openvpn/client/client.ovpn Insert certificate file\n{ echo \u0026#34;\u0026lt;cert\u0026gt;\u0026#34; awk \u0026#39;/BEGIN/,/END/\u0026#39; /etc/openvpn/easy-rsa/3/pki/issued/$CLIENT_NAME.crt echo \u0026#34;\u0026lt;/cert\u0026gt;\u0026#34; } | tee -a /etc/openvpn/client/client.ovpn Insert certificate key file\n{ echo \u0026#34;\u0026lt;key\u0026gt;\u0026#34; cat /etc/openvpn/easy-rsa/3/pki/private/$CLIENT_NAME.key echo \u0026#34;\u0026lt;/key\u0026gt;\u0026#34; } | tee -a /etc/openvpn/client/client.ovpn Insert tls-crypt key file\n{ echo \u0026#34;\u0026lt;tls-crypt\u0026gt;\u0026#34; awk \u0026#39;/BEGIN/,/END/\u0026#39; /etc/openvpn/cert/tls-crypt.key echo \u0026#34;\u0026lt;/tls-crypt\u0026gt;\u0026#34; } | tee -a /etc/openvpn/client/client.ovpn Now you complete generating client config in /etc/openvpn/client/client.ovpn\nSetup logging and logrotate First we create log directory\nmkdir -p /var/log/openvpn Then create logrotation policy\ncat \u0026lt;\u0026lt;EOF | tee /etc/logrotate.d/openvpn /var/log/openvpn/*.log { daily missingok rotate 7 compress delaycompress notifempty create 0640 root root sharedscripts postrotate /bin/systemctl reload openvpn-server@server \u0026gt; /dev/null 2\u0026gt;\u0026amp;1 || true endscript } EOF Make sure logrotate configuration\nlogrotate -d /etc/logrotate.d/openvpn Authentication using Linux PAM and OTP Here is sample SElinux policy for OpenVPN with Google Authenticator\ncat \u0026lt;\u0026lt;EOF | tee openvpn-custom.te module openvpn-custom 1.0; require { type openvpn_t; type user_home_t; type auth_home_t; type user_home_dir_t; class file { create getattr open read rename setattr unlink write }; class dir { read write add_name remove_name }; } #============= openvpn_t ============== # Allow openvpn to access auth_home_t files allow openvpn_t auth_home_t:file { open read unlink }; # Allow openvpn to access user_home_t directories and files allow openvpn_t user_home_dir_t:dir { read write add_name remove_name }; allow openvpn_t user_home_t:dir { read write add_name remove_name }; allow openvpn_t user_home_t:file { create getattr open read rename setattr unlink write }; EOF checkmodule -M -m -o openvpn-custom.mod openvpn-custom.te semodule_package -o openvpn-custom.pp -m openvpn-custom.mod semodule -i openvpn-custom.pp Add context for custom home directory\nsemanage fcontext -a -t auth_home_t /home/lab.tworty.id/[^/]+/\\.google_authenticator semanage fcontext -a -t auth_home_t /home/lab.tworty.id/[^/]+/\\.google_authenticator~ restorecon -rv /home Installing package and setup PAM policy\ndnf install -y google-authenticator qrencode cat \u0026lt;\u0026lt;EOF | tee /etc/pam.d/openvpn auth required pam_google_authenticator.so forward_pass auth include system-auth account include system-auth password include system-auth session include system-auth EOF Replace new server configuration with this values\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/openvpn/server/server.conf port 1194 proto udp dev tun user nobody group nobody persist-key persist-tun keepalive 10 120 topology subnet server 10.79.83.0 255.255.255.0 push \u0026#34;route 10.79.80.0 255.255.255.0\u0026#34; push \u0026#34;dhcp-option DNS 10.79.80.254\u0026#34; push \u0026#34;dhcp-option DOMAIN lab.tworty.id\u0026#34; client-config-dir /etc/openvpn/client/ccd ifconfig-pool-persist /etc/openvpn/client/persistent-ip.list dh none ecdh-curve prime256v1 auth SHA256 cipher AES-128-GCM data-ciphers AES-128-GCM ca /etc/openvpn/cert/ca.crt cert /etc/openvpn/cert/jumpserver.crt key /etc/openvpn/cert/jumpserver.key crl-verify /etc/openvpn/cert/crl.pem tls-crypt /etc/openvpn/cert/tls-crypt.key tls-server tls-version-min 1.2 tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256 status /var/log/openvpn/status.log log-append /var/log/openvpn/openvpn.log plugin /usr/lib64/openvpn/plugins/openvpn-plugin-auth-pam.so openvpn username-as-common-name duplicate-cn reneg-sec 0 verb 3 EOF Update openvpn systemd service file, based on this issue and restart service\ncp -r /usr/lib/systemd/system/openvpn-server@.service /etc/systemd/system sed -i \u0026#39;s/^ProtectHome=true/ProtectHome=false/\u0026#39; /etc/systemd/system/openvpn-server@.service systemctl daemon-reload systemctl restart openvpn-server@server.service Add this configuration to check when create new user or user login must create multi factor authenticator\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/skel/.bashrc if [ ! -f ~/.google_authenticator ]; then echo \u0026#34;Google Authenticator configuration file not found.\u0026#34; echo \u0026#34;Setting up Google Authenticator...\u0026#34; if google-authenticator -t -d -f -r 3 -R 30 -W; then echo \u0026#34;Google Authenticator has been set up successfully. Please scan the QR code using your authenticator app.\u0026#34; else echo \u0026#34;Error generating the QR code. Please check if google-authenticator is installed correctly and try again.\u0026#34; exit 1 fi else echo \u0026#34;Google Authenticator has already been set up.\u0026#34; fi EOF You can test google-authenticator PAM is working or not, you can use pamtester. You need to enter OTP.\ndnf install -y pamtester pamtester openvpn viki authenticate Generate Client Certificate and Client Configuration YOUR_IP=$(curl -s ipinfo.io/ip) CLIENT_NAME=client2fa cat \u0026lt;\u0026lt;EOF | tee /etc/openvpn/client/client2fa.ovpn client proto udp explicit-exit-notify remote $YOUR_IP 1194 dev tun resolv-retry infinite nobind persist-key persist-tun remote-cert-tls server verify-x509-name jumpserver name auth SHA256 auth-nocache auth-user-pass cipher AES-128-GCM tls-client tls-version-min 1.2 tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256 reneg-sec 0 verb 3 EOF cd /etc/openvpn/easy-rsa/3 ./easyrsa --batch build-client-full \u0026#34;$CLIENT_NAME\u0026#34; nopass Insert cerfificate authority file\n{ echo \u0026#34;\u0026lt;ca\u0026gt;\u0026#34; cat /etc/openvpn/server/ca.crt echo \u0026#34;\u0026lt;/ca\u0026gt;\u0026#34; } | tee -a /etc/openvpn/client/client2fa.ovpn Insert certificate file\n{ echo \u0026#34;\u0026lt;cert\u0026gt;\u0026#34; awk \u0026#39;/BEGIN/,/END/\u0026#39; /etc/openvpn/easy-rsa/3/pki/issued/$CLIENT_NAME.crt echo \u0026#34;\u0026lt;/cert\u0026gt;\u0026#34; } | tee -a /etc/openvpn/client/client2fa.ovpn Insert certificate key file\n{ echo \u0026#34;\u0026lt;key\u0026gt;\u0026#34; cat /etc/openvpn/easy-rsa/3/pki/private/$CLIENT_NAME.key echo \u0026#34;\u0026lt;/key\u0026gt;\u0026#34; } | tee -a /etc/openvpn/client/client2fa.ovpn Now you complete generating client config in /etc/openvpn/client/client2fa.ovpn\nClient Set up Sent /etc/openvpn/client/client.ovpn file to the user.\nReferences https://github.com/google/google-authenticator-libpam https://www.digitalocean.com/community/tutorials/how-to-set-up-and-configure-an-openvpn-server-on-centos-8 https://perfecto25.medium.com/openvpn-community-2fa-with-google-authenticator-4f2a7cb08128\n- https://www.ipfire.org/docs/configuration/services/openvpn/extensions/plugins/auth-pam https://build.openvpn.net/doxygen/group__tls__crypt.html#details https://medium.com/@pellabeuf/exploring-diffie-hellman-and-elliptic-curve-diffie-hellman-key-exchange-6b6d882933c2e ","permalink":"http://vikipranata.is-a.dev/posts/linux-openvpn-server/","summary":"\u003ch2 id=\"installing-package-dependencies\"\u003e\u003cem\u003eInstalling package dependencies\u003c/em\u003e\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ednf install -y epel-release\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ednf install -y openvpn openssl easy-rsa policycoreutils-python-utils \n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"creating-and-setting-up-an-openvpn-server-certificates\"\u003e\u003cem\u003eCreating and Setting Up an OpenVPN Server Certificates\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eSet of environment variable for easy_rsa\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ecat \u003cspan class=\"s\"\u003e\u0026lt;\u0026lt;EOF | tee vars\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eset_var EASYRSA_ALGO ec\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eset_var EASYRSA_CURVE prime256v1\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eset_var EASYRSA_CA_EXPIRE 3650\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eset_var EASYRSA_CERT_EXPIRE 3650\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eset_var EASYRSA_CRL_DAYS 3650\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eEOF\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThen execute \u003ccode\u003e./easyresa init-pki\u003c/code\u003e\u003cbr\u003e\n\u003cimg alt=\"init-pki\" loading=\"lazy\" src=\"/posts/linux-openvpn-server/uploads/init-pki.webp\"\u003e\u003c/p\u003e\n\u003cp\u003eCreate certificate authority (CA) and sever certificate.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eSERVER_CN\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003ejumpserver\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eSERVER_NAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003ejumpserver\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e./easyrsa --batch --req-cn\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$SERVER_CN\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e build-ca nopass\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e./easyrsa --batch build-server-full \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$SERVER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e nopass\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e./easyrsa gen-crl\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cimg alt=\"create-ca-cert\" loading=\"lazy\" src=\"/posts/linux-openvpn-server/uploads/create-ca-cert.webp\"\u003e\u003c/p\u003e","title":"Linux OpenVPN Server with Google Authenticators"},{"content":"After we Setup Active Directory Server in Windows Server 2022 now we can use this LDAP/AD User to login to server environments.\nJoin Active Directory and Configure to Login with User AD Installing packages\ndnf install -y realmd oddjob oddjob-mkhomedir sssd adcli samba-common-tools Set dns server to AD server\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/resolv.conf search lab.homelab.is-a.dev nameserver 10.79.80.3 EOF Discover to AD server\nrealm discover lab.homelab.is-a.dev Joining linux host to AD server\nrealm join lab.homelab.is-a.dev -U Administrator Adjust configuration in /etc/sssd/sssd.conf file\n[sssd] domains = lab.homelab.is-a.dev config_file_version = 2 services = nss, pam [domain/lab.homelab.is-a.dev] default_shell = /bin/bash krb5_store_password_if_offline = True cache_credentials = True krb5_realm = lab.homelab.is-a.dev realmd_tags = manages-system joined-with-adcli id_provider = ad fallback_homedir = /home/%d/%u ad_domain = lab.homelab.is-a.dev use_fully_qualified_names = False ldap_id_mapping = True access_provider = simple then restart it\nsystemctl restart sssd.service Setup sudo config for AD User\ncat \u0026lt;\u0026lt;EOF | tee /etc/sudoers.d/ad-users %administrators ALL=(ALL) ALL %operations\\ team ALL=(ALL) ALL EOF Maybe you can restrics ssh access with adding this configuration\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/ssh/sshd_config Allowgroups administrators \u0026#34;operations team\u0026#34; cloud-admin EOF systemctl restart sshd.service KDC has no support for encryption type issue Cek configuration file in /etc/krb5.conf then adjust this value\ndefault_realm = LAB.HOMELAB.IS-A.DEV default_tgs_enctypes = arcfour-hmac-md5 des-cbc-crc des-cbc-md5 default_tkt_enctypes = arcfour-hmac-md5 des-cbc-crc des-cbc-md5 Reference:\nhttps://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/system-level_authentication_guide/sssd https://discussion.fedoraproject.org/t/kdc-has-no-support-for-encryption-type/77540/3 ","permalink":"http://vikipranata.is-a.dev/posts/linux-join-ad/","summary":"\u003cp\u003eAfter we \u003ca href=\"../setup-active-directory-server\"\u003eSetup Active Directory Server in Windows Server 2022\u003c/a\u003e now we can use this LDAP/AD User to login to server environments.\u003c/p\u003e\n\u003ch2 id=\"join-active-directory-and-configure-to-login-with-user-ad\"\u003eJoin Active Directory and Configure to Login with User AD\u003c/h2\u003e\n\u003cp\u003eInstalling packages\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ednf install -y realmd oddjob oddjob-mkhomedir sssd adcli samba-common-tools\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSet dns server to AD server\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ecat \u003cspan class=\"s\"\u003e\u0026lt;\u0026lt;EOF | tee -a /etc/resolv.conf\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003esearch lab.homelab.is-a.dev\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003enameserver 10.79.80.3\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003eEOF\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eDiscover to AD server\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003erealm discover lab.homelab.is-a.dev\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cimg alt=\"realm-discover\" loading=\"lazy\" src=\"/posts/linux-join-ad/uploads/realm-discover.webp\"\u003e\u003c/p\u003e\n\u003cp\u003eJoining linux host to AD server\u003c/p\u003e","title":"Linux Join Active Directory"},{"content":" This pages is work in progress\nSetup Active Directory Domain Services Setup finish and server will be restart.\nCreate Organizational Unit and Security Group Create User and Assign to Security Group Add a Domain Controller to an Existing Domain Setup finish and server will be restart.\n","permalink":"http://vikipranata.is-a.dev/posts/setup-active-directory-server/","summary":"\u003cblockquote\u003e\n\u003cp\u003eThis pages is work in progress\u003c/p\u003e\u003c/blockquote\u003e\n\u003ch2 id=\"setup-active-directory-domain-services\"\u003eSetup Active Directory Domain Services\u003c/h2\u003e\n\u003cp\u003e\u003cimg alt=\"Step 1\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/1.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 2\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/2.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 3\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/3.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 4\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/4.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 5\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/5.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 6\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/6.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 7\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/7.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 8\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/8.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 9\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/9.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 10\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/10.webp\"\u003e\u003c/p\u003e\n\u003cp\u003eSetup finish and server will be restart.\u003c/p\u003e\n\u003ch2 id=\"create-organizational-unit-and-security-group\"\u003eCreate Organizational Unit and Security Group\u003c/h2\u003e\n\u003cp\u003e\u003cimg alt=\"Step 11\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/11.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 12\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/12.webp\"\u003e\u003c/p\u003e\n\u003ch2 id=\"create-user-and-assign-to-security-group\"\u003eCreate User and Assign to Security Group\u003c/h2\u003e\n\u003cp\u003e\u003cimg alt=\"Step 13\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/13.webp\"\u003e \u003cbr\u003e\n\u003cimg alt=\"Step 14\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/14.webp\"\u003e\u003c/p\u003e\n\u003ch2 id=\"add-a-domain-controller-to-an-existing-domain\"\u003eAdd a Domain Controller to an Existing Domain\u003c/h2\u003e\n\u003cp\u003e\u003cimg alt=\"Step 15\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/15.webp\"\u003e\u003cbr\u003e\n\u003cimg alt=\"Step 16\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/16.webp\"\u003e\u003cbr\u003e\n\u003cimg alt=\"Step 17\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/17.webp\"\u003e\u003cbr\u003e\n\u003cimg alt=\"Step 18\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/18.webp\"\u003e\u003cbr\u003e\n\u003cimg alt=\"Step 19\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/19.webp\"\u003e\u003cbr\u003e\n\u003cimg alt=\"Step 20\" loading=\"lazy\" src=\"/posts/setup-active-directory-server/uploads/20.webp\"\u003e\u003c/p\u003e\n\u003cp\u003eSetup finish and server will be restart.\u003c/p\u003e","title":"Setup Active Directory Server with Windows Server 2022"},{"content":"Configure Linux NAT Forwarding Enable kernel parameter\necho \u0026#39;net.ipv4.ip_forward = 1\u0026#39; \u0026gt; /etc/sysctl.d/99-forwarding.conf sysctl --system Setup firewalld configuration\nnmcli connection migrate nmcli connection modify eth0 connection.zone public nmcli device modify eth0 connection.zone public nmcli connection modify eth1 connection.zone internal nmcli device modify eth1 connection.zone internal firewall-cmd --permanent --zone=public --add-masquerade firewall-cmd --permanent --new-policy NAT-int-to-ext firewall-cmd --permanent --policy NAT-int-to-ext --add-ingress-zone internal firewall-cmd --permanent --policy NAT-int-to-ext --add-egress-zone public firewall-cmd --permanent --policy NAT-int-to-ext --set-target ACCEPT firewall-cmd --reload Special case for Proxmox Virtual Environment\nauto vmbr0 iface vmbr0 inet manual bridge-ports eth0 bridge-stp off bridge-fd 0 post-up echo 1 \u0026gt; /proc/sys/net/ipv4/conf/vmbr0/forwarding post-down echo 0 \u0026gt; /proc/sys/net/ipv4/conf/vmbr0/forwarding #Public Network Configure linux simple DNS server for lab purposes Installing package dependencies\ndnf install -y dnsmasq Setup dnsmasq upstream forward server\ncat \u0026lt;\u0026lt;EOF | tee -a /etc/dnsmasq.conf server=1.1.1.1 server=1.0.0.1 EOF Setup dnsmasq to rewrite dns request\ncat \u0026lt;\u0026lt;EOF | tee /etc/dnsmasq.d/lab.homelab.is-a.dev.conf address=/homelab.is-a.dev/10.79.80.3 cname=blog.homelab.is-a.dev,homelab.is-a.dev EOF References https://wiki.archlinux.org/title/Firewalld#NAT_masquerade https://serverfault.com/questions/1005682/proxmox-ipv4-inward-routing-not-working ","permalink":"http://vikipranata.is-a.dev/posts/linux-networking/","summary":"\u003ch2 id=\"configure-linux-nat-forwarding\"\u003e\u003cem\u003eConfigure Linux NAT Forwarding\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eEnable kernel parameter\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;net.ipv4.ip_forward = 1\u0026#39;\u003c/span\u003e \u0026gt; /etc/sysctl.d/99-forwarding.conf\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003esysctl --system\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSetup firewalld configuration\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003enmcli connection migrate\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003enmcli connection modify eth0 connection.zone public\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003enmcli device modify eth0 connection.zone public\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003enmcli connection modify eth1 connection.zone internal\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003enmcli device modify eth1 connection.zone internal\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003efirewall-cmd --permanent --zone\u003cspan class=\"o\"\u003e=\u003c/span\u003epublic --add-masquerade\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003efirewall-cmd --permanent --new-policy NAT-int-to-ext\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003efirewall-cmd --permanent --policy NAT-int-to-ext --add-ingress-zone internal\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003efirewall-cmd --permanent --policy NAT-int-to-ext --add-egress-zone public\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003efirewall-cmd --permanent --policy NAT-int-to-ext --set-target ACCEPT\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003efirewall-cmd --reload\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSpecial case for Proxmox Virtual Environment\u003c/p\u003e","title":"Linux Networking Self Notes"},{"content":"Expand Space without Rebooting VM Current usage disk with lsblk command And here is LVM partition table\nFirst step we need grow partition in /dev/sda3, install package growpart first\ndnf install -y cloud-utils-growpart Then we resize partition with this command\ngrowpart /dev/sda 3 See the different before and after, if you don\u0026rsquo;t use LVM partition scheme just execute resize2fs /dev/sda3 to full fill the partition.\npvresize /dev/sda3 pvs vgs Now you\u0026rsquo;re ready to extent the LVM partition\nlvresize --extents +100%FREE --resizefs /dev/vg0/root If you just resize partition only 20%, this sample command\nlvresize --size +20G --resizefs /dev/vg0/root Expand Space without Rebooting VM in ESXi Guest Host Re-scan the SCSI Bus to Add a SCSI Device Without rebooting the VM using the following command\nFirst, check the name(s) of your scsi devices.\nls /sys/class/scsi_device/ Then rescan the scsi bus. Below you can replace the \u0026lsquo;0\\:0\\:0\\:0′ with the actual scsi bus name found with the previous command. Each colon is prefixed with a slash, which is what makes it look weird.\necho 1 \u0026gt; /sys/class/scsi_device/0\\:0\\:0\\:0/device/rescan Resize disk but no space left if there is a condition where there is no space left on the block device, you can use the following command:\nmount -o size=10M,rw,nodev,nosuid -t tmpfs tmpfs /tmp then do the same workaround to resize the disk\nReferences https://access.redhat.com/solutions/5540131 https://www.redhat.com/sysadmin/resize-lvm-simple https://communities.vmware.com/t5/vSphere-Hypervisor-Discussions/Expand-Space-without-Rebooting-VM/td-p/923809 ","permalink":"http://vikipranata.is-a.dev/posts/linux-resize-disk/","summary":"\u003ch2 id=\"expand-space-without-rebooting-vm\"\u003e\u003cem\u003eExpand Space without Rebooting VM\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003eCurrent usage disk with \u003ccode\u003elsblk\u003c/code\u003e command\n\u003cimg alt=\"disk-overview\" loading=\"lazy\" src=\"/posts/linux-resize-disk/uploads/disk-overview.webp\"\u003e\u003c/p\u003e\n\u003cp\u003eAnd here is LVM partition table\u003cbr\u003e\n\u003cimg alt=\"lvm-overview\" loading=\"lazy\" src=\"/posts/linux-resize-disk/uploads/lvm-overview.webp\"\u003e\u003c/p\u003e\n\u003cp\u003eFirst step we need grow partition in \u003ccode\u003e/dev/sda3\u003c/code\u003e, install package growpart first\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ednf install -y cloud-utils-growpart\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThen we resize partition with this command\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egrowpart /dev/sda \u003cspan class=\"m\"\u003e3\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cimg alt=\"growpart-result\" loading=\"lazy\" src=\"/posts/linux-resize-disk/uploads/growpart-result.webp\"\u003e\u003c/p\u003e\n\u003cp\u003eSee the different before and after, if you don\u0026rsquo;t use LVM partition scheme just execute \u003ccode\u003eresize2fs /dev/sda3\u003c/code\u003e to full fill the partition.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003epvresize /dev/sda3\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003epvs\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003evgs\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cimg alt=\"pvresize-result\" loading=\"lazy\" src=\"/posts/linux-resize-disk/uploads/pvresize-result.webp\"\u003e\u003c/p\u003e","title":"Linux Resize Disk"},{"content":"Hi, 👋\nMy name is Berto Viki Satria Pranata, also known as Viki Pranata for short. I am a dedicated IT professional with experience in Information Technology since 2015, specializing in systems administration, cloud engineering, and IT architecture. I am skilled in managing complex IT infrastructures and implementing solutions with many technology stacks, Utilizing various automation tools and best practices in production environments.\nTo view my full profile, please connect with me on LinkedIn.\nAchievment Certified Kubernetes Administrator, issued on 10 January 2023 Certified OpenStack Administrator, issued on 12 January 2023 Mikrotik Certified Network Associate, issued on 20 July 2023 Training Coursera, Google Cloud Platform Fundamentals: Core Infrastructure, 21 June 2020 Coursera, Essential Cloud Infrastructure: Foundation, 06 July 2020 Digital Talent Scholarship - Kominfo, PROA - Container, Kubernetes \u0026amp; OpenShift, 30 April 2021 Adinusa, Linux System Administration, 08 July 2021 Adinusa, Docker For DevOps, 26 July 2021 Adinusa, Ansible Administration, 10 August 2021 Adinusa, Kubernetes Administration, 18 August 2021 Adinusa, Ceph Administration, 23 August 2021 Adinusa, OpenStack Administration, 31 August 2021 Dicoding, Belajar Dasar-Dasar DevOps, 23 January 2023 Dicoding, Belajar Jaringan Komputer untuk Pemula, 24 January 2023 Dicoding, Cloud Practitioner Essentials (Belajar Dasar AWS Cloud), 07 Maret 2023 Dicoding, Menjadi Linux System Administrator, 28 August 2023 Dicoding, Belajar Implementasi CI/CD, 31 August 2023 Continue to be a lifelong learner. ","permalink":"http://vikipranata.is-a.dev/about/","summary":"\u003cp\u003eHi, 👋\u003cbr\u003e\nMy name is Berto Viki Satria Pranata, also known as Viki Pranata for short. I am a dedicated IT professional with experience in Information Technology since 2015, specializing in systems administration, cloud engineering, and IT architecture. I am skilled in managing complex IT infrastructures and implementing solutions with many technology stacks, Utilizing various automation tools and best practices in production environments.\u003cbr\u003e\nTo view my full profile, please connect with me on \u003ca href=\"https://www.linkedin.com/in/vikipranata\"\u003eLinkedIn\u003c/a\u003e.\u003c/p\u003e","title":"About me"}]