I've been experimenting with OpenClaw for a few weeks, and I'm excited about what it can do for me. OpenClaw is an open-source AI-agent framework that lets agents interact with tools and skills while I chat with them on channels like Telegram or Discord. The best part is that it can act as a personal assistant, handling day-to-day activities.

In this post, I will go over how I set up OpenClaw on a Linux VM, connected it to Telegram, and gave it access to Google services like Calendar, Gmail, and Tasks. Since OpenClaw by itself comes with a few security risks, it is best to have an architecture whereby the AI agent does not get too much information/data about my credentials that I provide. The goal is to create an AI agent that can interact with my tools and accounts without exposing my main credentials.

Installing OpenClaw

First, follow the official installation guide:

https://docs.openclaw.ai/start/getting-started

For my setup:

• Created a Proxmox virtual machine and installed Ubuntu Server (ubuntu-22.04.5-live-server-amd64.iso) on it

• I am using gpt-3.5-codex-medium as my LLM model, because I have one month free subscription for OpenAI's Plus plan. This gives me an ample amount of credits to use for OpenClaw.

• I added a Telegram channel for OpenClaw, the main area where I will be chatting with it. This can be created using the BotFather bot and calling /newbot. After creating it, it will provide you a token as shown below, where you can input it during the OpenClaw setup.

  • The first chat will probably be blocked by the gateway. Run the following command to authorize the channel: openclaw pairing approve telegram ABCDEFG

Once installed, you should be able to start the OpenClaw agent and interact with it through the CLI or Telegram. The first time it's being run, it will ask questions about you and itself, where you can provide information about yourself.

After interacting with the AI agent. It will write itself into AGENTS.md, MEMORY.md, SOUL.md, USER.md, so that next time when it runs on a new session again, it can be loaded back in the context and will seem as though it remembers things about you. You can also manually edit these files to your liking as well.

Configuring Git for the Agent

Some OpenClaw workflows require Git access (for example, committing code changes or interacting with repositories).

Configure the Git identity used by the agent:

git config --global user.name "name"
git config --global user.email "name@gmail.com"

This ensures that any commits created by the agent are properly attributed.

Giving OpenClaw Access to Google Services

One of the more interesting things I wanted the agent to do was interact with:

• Gmail

• Google Calendar

• Google Tasks

Instead of exposing my personal Google account, I created a separate Gmail account dedicated to OpenClaw. It is free to execute things on its own Google account without having authorization to my personal Google account. This helps further reduce risk, and I will not be surprised to find out one day that all my emails were deleted by AI.

I will be installing the gog skill on Clawdhub.ai. OpenClaw allows you to extend the agent by creating skills. This skill allows me to connect to Gmail, Calendar, Drive and others. It requires the gogcli and oauth to be set up first.

Installing gog skill

Create the skill directory:

mkdir -p .openclaw/skills/gog

Then create the skill definition file:

vim .openclaw/skills/gog/SKILL.md

This file tells OpenClaw how to use the tool. Means the AI will read this skill, understand it and then execute tools such as gogcli. SKILL.md can be directly copied from the clawhub website here

Installing gogcli

https://github.com/steipete/gogcli

Download the binary:

wget https://github.com/steipete/gogcli/releases/download/v0.11.0/gogcli_0.11.0_linux_amd64.tar.gz

Extract it:

tar -xzf gogcli_0.11.0_linux_amd64.tar.gz

Move the binary into your path:

sudo mv gog /usr/local/bin/

Verify the installation:

gog --help

Creating Google OAuth Credentials

To allow gogcli to access Google services, we need to create OAuth credentials.

Open the Google Cloud Console credentials page:

https://console.cloud.google.com/apis/credentials

Then follow these steps:

1. Create a new project

2. Enable APIs

   • Gmail API

   • Calendar API

   • Tasks API

3. Configure the OAuth consent screen

4. Add your Gmail account as a test user

5. Create an OAuth client

Download the credentials file:

client_secret_....apps.googleusercontent.com.json

Authenticating gogcli

Once you have the OAuth credentials, store them using:

gog auth credentials ./client_secret_.apps.googleusercontent.com.json

Then authorize your Google account:

gog auth add name@gmail.com --services user --manual

You will be prompted for a keyring password. For simplicity, I used openclaw.

In SKILL.md, update the instructions so the agent knows how to run the tool properly. Running the tool in Linux requires the GOG_KEYRING_PASSWORD environment variable.

Notes

- Set `GOG_ACCOUNT=name@gmail.com` to avoid repeating `--account`.
- Always set `GOG_KEYRING_PASSWORD=openclaw`.
- Always add `--no-input` when executing commands.

What the Agent Can Do Now

After completing the setup, the OpenClaw agent can now:

• Read my Google Calendar

• Check emails

• View tasks

I then subscribed to the calendar of my main Google account. After giving my permission to read and write, I can tell my AI agent to add a schedule for plans and then every night, notify me of all the things I have in my calendar for tomorrow.

Here are the things we have done for now:

more t

proxmox host setup for GPU passthrough

1. Verify CPU supports hardware virtualization and IOMMU

   • might have to enable settings in the BIOS

   • mine was under Asus BIOS → CPU advanced settings → vmx virtualization technology

2. In the Proxmox host machine shell, add intel_iommu=on to /etc/default/grub

    # /etc/default/grub
   
    GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
   

3. update-grub and then sudo reboot to save settings

4. Verify IOMMU is turned on: dmesg | grep -e DMAR -e IOMMU

    dmesg | grep -e DMAR -e IOMMU
   
    [    0.007531] ACPI: DMAR 0x000000008EC3B3C0 000070 (v01 INTEL  KBL      00000001 INTL 00000001)
    [    0.007554] ACPI: Reserving DMAR table memory at [mem 0x8ec3b3c0-0x8ec3b42f]
    [    0.027442] DMAR: IOMMU enabled
    [    0.074297] DMAR: Host address width 39
    [    0.074298] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
    [    0.074303] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
    [    0.074305] DMAR: RMRR base: 0x0000008eaee000 end: 0x0000008eb0dfff
    [    0.074308] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
    [    0.074309] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
    [    0.074310] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
    [    0.075656] DMAR-IR: Enabled IRQ remapping in x2apic mode
    [    0.262942] DMAR: No ATSR found
    [    0.262943] DMAR: No SATC found
    [    0.262944] DMAR: dmar0: Using Queued invalidation
    [    0.263559] DMAR: Intel(R) Virtualization Technology for Directed I/O
   

5. we do not want the host machine to use the GPU, so we blacklist Nvidia drivers

    # /etc/modprobe.d/blacklist.conf
   
    blacklist nouveau
    blacklist nvidiafb
   

6. Find our GPU hardware device, lspci -nn

   We can see 01:00.0 and 01:00.1 here correspond to our graphics card, and lets get the vendor and device ID: 10de:1c02 and 10de:10f1

    lscpi -nn
   
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
    01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
   

7. Add the graphics card we want to pass through with the vendor and device IDs we found earlier and do a restart

    echo "options vfio-pci ids=10de:1c02,10de:10f1 disable_vga=1" > /etc/modprobe.d/vfio.conf
    update-initramfs -u
    sudo reboot
   

proxmox virtual machine setup for GPU passthrough

1. Most of the steps are similar to creating a normal virtual machine

2. When selecting a CPU type, be sure to use host instead of a virtual CPU, as it may not support certain instruction sets

3. Be sure to allocate enough memory for the ollma model that you are using

4. After the virtual machine is created, do not start it yet, go to hardware → add → PCI device and select our graphics card

   

5. Start our virtual machine, follow the installation, etc

6. Install GPU drivers in the virtual machine

    # running as su
   
    add-apt-repository ppa:graphics-drivers/ppa
    apt update
    apt upgrade
    ubuntu-drivers autoinstall
    echo "blacklist nouveau" > /etc/modprobe.d/blacklist-nouveau.conf
    echo "options nouveau modeset=0" >> /etc/modprobe.d/blacklist-nouveau.conf
    update-initramfs -u
    reboot
   

7. Check our GPU status

    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
    |-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  NVIDIA GeForce GTX 1060 3GB    Off |   00000000:01:00.0 Off |                  N/A |
    |  0%   41C    P8              7W /  180W |       5MiB /   3072MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
   
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |  No running processes found                                                             |
    +-----------------------------------------------------------------------------------------+
   

installing ollama

curl -fsSL https://ollama.com/install.sh | sh

ollama run deepseek-r1:14b

# test ollama inference
>>> what is 2 + 2
<think>
I see the question is asking for the sum of two plus two.

First, I'll identify the numbers involved in the addition.

Next, I'll add them together to find the total.

Finally, I'll present the final answer clearly.
</think>

Sure! Let's solve \(2 + 2\) step by step:

1. **Identify the Numbers:**
   - We have two numbers to add: 2 and 2.

2. **Add the Numbers:**
   \[
   2 + 2 = 4
   \]

3. **Final Answer:**
   \[
   \boxed{4}
   \]

Now we can run our model in the command line, we can then connect to a front end to use our model as well.

installing open-webui

# install docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh ./get-docker.sh
sudo usermod -aG docker $USER
logout

# pull open-webui container
docker pull ghcr.io/open-webui/open-webui:main

# run container on host network
docker run -d -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -v open-webui:/app/backend/data --network=host --name open-webui ghcr.io/open-webui/open-webui:main

# visit website
http://<host ip>:8080

• After visiting the website and creating an account, go to settings → connection, update the Ollama API connection to http://127.0.0.1:11434 (they are in the same virtual machine)

  

• We can ask any question like ChatGPT

  • 

    not the best hardware, seeing that it took about 6 minutes to generate a response, but it is something to try =)

overview

Here’s a brief overview of my internal network.

Storage server

192.168.1.10 is a mini pc designated as my main storage server, which I used to set up my network file-sharing server and Samba server. I decided to keep most of my setup here pretty simple with no virtualization and opt to run more critical and fundamental services.

The main services running here are my prometheus-grafana-loki monitoring stack which collects metrics and logs from other hosts for display. I also run Adguard here, which is my DNS resolver for my home network. Authentik is an identity provider that supports ODIC protocol for SSO sign-in to my internal applications. Vaultwarden serves as another password manager I use for authentication in my internal application when Authentik is not an option.

Here are some examples of my monitoring visualization:

proxmox server

192.168.1.11 is another mini pc running my proxmox operating system that can create virtual machines. Not a lot is done here on this physical machine.

application server

192.169.1.13 is a virtual machine running docker. Through docker-compose.yaml scripts I was able to set up my application services quickly. Some applications such as immich and hoarder rely on NFS storage from my storage server. Here I have also set up Caddy reverse proxy, which is the entry point for most of my internal applications. Incoming network request to my internal application is first routed to this reverse proxy and then to other machines through IP and port, or via docker network if on the same host.

Here are what some of the apps do:

• Immich: Google Photo alternative

• Paperless: stores documents

• Dashy: dashboard

• Gitea: git repository and docker image storage

• Hoarder: stores website links and web pages

• Outline: notion like a note-taking app

• Caddy: reverse proxy entry point for most apps

tailscale server

192.169.1.12 is another virtual machine that only runs Tailscale. Tailscale provides a VPN, allowing me to connect to my home network even if I am outside. My tailscale node here is configured as an exit node.

kubernetes cluster

192.168.1.2 is an old desktop pc running proxmox operating system for most of my testing builds. I can quickly start up and delete virtual machines easily. I also have Kubernetes clusters setup, running 1 master node and 2 worker nodes, right now mainly for experimental purposes only.

DNS connection

To connect to my internal applications easily, I have set up a DNS Type A record on Cloudflare. So instead of using the IP address and port to connect, I can use <app name>.wongandre.com to resolve the IP address of my Caddy reverse proxy.

dashy homepage

Here is a homepage of my services, which is grouped into different categories. Monitoring for metrics and logging related apps. User management for internal app identity management. Networking for Adguard DNS, router configs, and proxmox configs. Data management for storing personal data like images, documents, or repositories.

future plans 2025

some plans to improve my home lab I have in mind right now

• self-host my own large language model such as Ollama

• create a production Kubernetes environment and migrate/add services to it

• Infrastructure as Code

  • write scripts to automate provisioning proxmox VMs and LXC through Terraform

  • write scripts to automate installing configurations per instance through Ansible

• Add more storage with direct attached storage (DAS)

• Configure backup for my data

• add more apps!

I have been using Google Cloud to store my data such as photos, documents, and emails. I paid about $3 a month for the 100GB plan and recently it is about 80% full.

This prompted me to look for another solution: self-hosting a file server that I can access within my network. This will provide me with much more needed space than the 100GB I have now and also will provide external storage for my Proxmox and Kubernetes projects down the road.

One con of a self-hosted file server is that if the data is corrupted, we might not have a guaranteed way to recover it. This means a backup is crucial if the data stored is important (this can be looked into in the future). Other factors to consider would be power and security. Availability is not an issue for me as files may not be accessed that often.

Hardware

For the main server, I went with a Beelink Mini S12 Mini PC ($240). Here are some of the specifications:

• Intel Alder Lake N100 Processor

  • it is a low-cost processor with 4 cores and uses very low energy of 6W. Assuming we run it 24 hours in a month of 30 days, we only use about 6/1000 × 24 × 30 = 4.32 kWh/month, looking at the price of electricity right now is 32.57 cents/kWh, so theoretically we are paying about $1.407 a month to maintain our server. Seems not too bad.

• 16GB DDR4 RAM

  • 16GB RAM should be ample RAM for our server. The brand of the ram is not stated.

• 500GB SSD

  • 500GB should be enough to run the server’s operating system and host a few applications.

• Wi-Fi 6 and Bluetooth 5.2

  • I will be using an ethernet cable plugged directly into my router, so Wi-Fi is not that important. Bluetooth can be useful if I want to connect a keyboard to it. Most of the time I will communicate through SSH, so this is not really needed.

For my file storage, I got a 2TB Samsung 870 EVO 2.5-inch SSD ($240).

Installation

I installed an Ubuntu server of version 22.04.5 LTS. Installation instructions can be found on their website.

I then freeze the version of Linux and do an apt update and upgrade

# prevent linux from upgrading itself when it does an upgrade
sudo apt-mark hold $(uname -r)

# upgrade packages
sudo apt update
sudo apt list --upgradable
sudo apt upgrade

Static IP configuration

By default, the network automatically uses DHCP to get an IP address from the router. I want mine to be fixed 192.168.1.10 for easy reference. Let’s configure a static IP address for it.

Before changing anything in my /etc/netplan/50-cloud-init.yaml file, the code was:

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        enp1s0:
            dhcp4: true
    version: 2

After adding the IP address that we want to be static:

network:
    ethernets:
        enp1s0:
            dhcp4: no
            addresses: [192.168.1.10/24]
            routes:
                - to: default
                  via: 192.168.1.254
            nameservers:
                addresses: [1.1.1.1,8.8.8.8,8.8.8.4]
    version: 2

dhcp4 is set to no, addresses are set to my static IP 192.168.1.10/24, routes indicate the internal gateway IP address which is 192.168.1.254. Nameservers are the DNS servers for name resolution, 1.1.1.1 for Cloudflare DNS and 8.8.8.8 for Google DNS.

As seen from the comments above, we need to disable the cloud-init by Ubuntu which will override this configuration at every reboot. Add to /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg, network: {config: disabled}.

Run sudo netplan try to apply the changes

Bluetooth configuration

Since our device comes with Bluetooth, it would be great if it can pair with my wireless keyboard.

We need to install bluez package

sudo apt install bluez -y

Entering the command bluetoothctl enters to a mini program.show shows no default controller available, which means my Bluetooth was not being detected.

Looking at the driver’s message by running: sudo dmesg|egrep -i 'blue|firm', there is an error bluetooth hci0: Direct firmware load for intel/ibt-0040-1050.sfi failed with error -2 which means that the Bluetooth firmware was not being loaded. Turns out it was not included in the /lib/firmware/intel/ directory.

To resolve this, we need to download it from elsewhere and do a reboot:

cd /lib/firmware/intel/
sudo wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/ibt-0040-1050.sfi
sudo wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/intel/ibt-0040-1050.ddc
reboot

Entering bluetoothctl, we can scan on to scan for nearby devices to pair. pair <address> to initiate the pair. Then scan off to turn off Bluetooth scanning again.

To make it automatically pair upon turning on device, we install the bluez-tool and run the daemon:

sudo apt install bluez-tools -y
bt-agent --capability=NoInputNoOutput -d

--capability=NoInputNoOutput will prevent the agent for asking a confirmation for connection.

New drive configuration

We can take a look at our storage and initialize the SSD we have just added

# looking at the ubuntu storage
sudo fdisk -l /dev/sda
#Disk /dev/sda: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
#Disk model: 512GB SSD
#Units: sectors of 1 * 512 = 512 bytes
#Sector size (logical/physical): 512 bytes / 512 bytes
#I/O size (minimum/optimal): 512 bytes / 512 bytes
#Disklabel type: gpt
#Disk identifier: CA61C6D8-AF70-47CD-B813-C8B689C730AC

#Device       Start        End   Sectors   Size Type
#/dev/sda1     2048    2203647   2201600     1G EFI System
#/dev/sda2  2203648    6397951   4194304     2G Linux filesystem
#/dev/sda3  6397952 1000212479 993814528 473.9G Linux filesystem

# looking at the 2T ssd we added
sudo fdisk -l /dev/sdb
#Disk /dev/sdb: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
#Disk model: Samsung SSD 870
#Units: sectors of 1 * 512 = 512 bytes
#Sector size (logical/physical): 512 bytes / 512 bytes
#I/O size (minimum/optimal): 512 bytes / 512 bytes

# create a Linux LVM partition using gdisk
sudo gdisk /dev/sdb
n                  # new partition
1                  # partition number
2048 <enter>       # start of sector
3907029134 <enter> # end of sector
8e00               # code for Linux LVM
# result
# Number  Start (sector)    End (sector)  Size       Code  Name
#  1            2048      3907029134   1.8 TiB     8E00  Linux LVM
w                  # write changes
# Do you want to proceed? (Y/N): Y
# OK; writing new GUID partition table (GPT) to /dev/sdb.
# The operation has completed successfully.

Once done, we can use our newly formatted partition and make it into an LVM physical volume

sudo pvcreate /dev/sdb1
# Physical volume "/dev/sdb1" successfully created.

# view our new physical volume
sudo pvdisplay /dev/sdb1
#  "/dev/sdb1" is a new physical volume of "<1.82 TiB"
#  --- NEW Physical volume ---
#  PV Name               /dev/sdb1
#  VG Name
#  PV Size               <1.82 TiB
#  Allocatable           NO
#  PE Size               0
#  Total PE              0
#  Free PE               0
#  Allocated PE          0
#  PV UUID               ZFZ04d-JmAS-lbgt-yF5a-s56v-vXue-sO2sfD

Next, we can add our physical volume to our newly created volume group to host a bunch of logical volumes that we will create later.

sudo vgcreate storage /dev/sdb1
# Volume group "storage" successfully created

sudo vgdisplay storage
#  --- Volume group ---
#  VG Name               storage
#  System ID
#  Format                lvm2
#  Metadata Areas        1
#  Metadata Sequence No  1
#  VG Access             read/write
#  VG Status             resizable
#  MAX LV                0
#  Cur LV                0
#  Open LV               0
#  Max PV                0
#  Cur PV                1
#  Act PV                1
#  VG Size               <1.82 TiB
#  PE Size               4.00 MiB
#  Total PE              476931
#  Alloc PE / Size       0 / 0
#  Free  PE / Size       476931 / <1.82 TiB
#  VG UUID               P6S18y-5mK1-e1eB-c6CS-wiWx-Tkkl-ocmf5T

Next, we can create a bunch of logical volumes, make the filesystem for each logical volume, and then mount them to our Linux

sudo lvcreate -n files -L 100G storage
# Logical volume "files" created.
sudo lvcreate -n photos -L 100G storage
# Logical volume "photos" created.

# check our logical volume
ls /dev/mapper/storage*
# /dev/mapper/storage-files  /dev/mapper/storage-photos

# make filesystem on logical volume
sudo mkfs -t ext4 /dev/mapper/storage-files
sudo mkfs -t ext4 /dev/mapper/storage-photos

# make mount directories
sudo mkdir /mnt/files
sudo mkdir /mnt/photos

# mount logical volumes
sudo mount /dev/mapper/storage-files /mnt/files/
sudo mount /dev/mapper/storage-photos /mnt/photos/

# view result
df -h
# Filesystem                         Size  Used Avail Use% Mounted on
# /dev/mapper/ubuntu--vg-ubuntu--lv   98G  7.2G   86G   8% /
# /dev/sda2                          2.0G  126M  1.7G   7% /boot
# /dev/sda1                          1.1G  6.1M  1.1G   1% /boot/efi
# /dev/mapper/storage-files           98G   24K   93G   1% /mnt/files
# /dev/mapper/storage-photos          98G   24K   93G   1% /mnt/photos

The good thing about logical volume is we can always add space if we slowly start running out. To resize, we can run lvextend and resize2fs -p.

To make sure our logical volumes are mounted automatically when we boot into our system, update the fstab file as so:

# /etc/fstab: static file system information.
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# other mounts ...
# my own mounts
/dev/mapper/storage-files /mnt/files ext4 defaults 0 2
/dev/mapper/storage-photos /mnt/photos ext4 defaults 0 2

The first 2 columns are the file system, and the mount point paths, the 3rd column is the type of file system. The 4th column is additional options that we set to defaults. The 5th column is for the dump command that we will disable. The 6th column sets the order in which the check disk commands run, 0 for no checking, 1 for the root drive, and 2 for other drives.

SSH configuration

To make login easier, we can do password-less authentication during ssh.

On our client's PC to connect, create our public and private RSA key pair via ssh-keygen. Follow the instructions shown by pressing enter.

Then copy the public key over to the server:

ssh-copy-id -i ~/.ssh/id_rsa.pub andre@192.168.1.10

Now we can ssh without entering the user password every time.

Network file server configuration

install the required server app and edit configurations related to the nfs server

# install the nfs server
apt install nfs-kernel-server

# edit /etc/exports
# assuming that we only have 1 ip client from 192.168.1.106
/mnt/files 192.168.1.106(rw,sync,no_subtree_check)
/mnt/photos 192.168.1.106(rw,sync,no_subtree_check)

# export the changes
sudo exportfs -rav

# update nfs-kernel-server settings
vim /etc/default/nfs-kernel-server
# change to RPCVCGSSDOPTS="-N 2 -N 3"
# this removes version 2 and 3 of nfs and only runs nfs version 4


# restart the server
sudo systemctl restart nfs-kernel-server

• configure the firewall for access

    sudo ufw allow from 192.168.1.106 to any port 2049
  
    sudo ufw enable
  
    # check firewall status
    sudo ufw status
    # Status: active
    # To                         Action      From
    # --                         ------      ----
    # 22/tcp                     ALLOW       Anywhere
    # 2049                       ALLOW       192.168.1.106
    # 22/tcp (v6)                ALLOW       Anywhere (v6)
  

• Ensure that the directory under /mnt/files and /mnt/photos are the ownership of the start user id of 1000

    id 1000
    # uid=1000(andre) gid=1000(andre) groups=1000(andre),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),110(lxd)
  
    # update ownership of dir
    sudo chown -R andre:andre /mnt/files
    sudo chown -R andre:andre /mnt/photos
  

• On the client PC that we want to connect to our nfs, we can install the nfs-common package

    sudo apt install nfs-common
  
    # try to mount nfs on client machine
    sudo mount -t nfs -vvv 192.168.1.10:/mnt/files /mnt/files
  
    # show mounts
    mount -t nfs4
    #192.168.1.10:/mnt/files on /mnt/files type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.106,local_lock=none,addr=192.168.1.10)
  
    # once mounted we can write to the nfs directory
    # should be able to write if client user id is also 1000
    echo "hello world" > /mnt/files/testfile
  

• To automatically mount the NFS every time the client machine starts up, we can update the /etc/fstab file

    # /etc/fstab
    192.168.1.10:/mnt/files /mnt/files nfs bg,rsize=8192,wsize=8192 0 0
  

  Where bg is an option to run the mount in the background later if the mount fails. rsize and wsize sets the buffer size for reading and writing, by default it is 1024.

Samba file server configuration

• create a new logical volume for samba and mount directory

    # new logical volume
    sudo lvcreate -n sambashare -L 200G storage
    sudo mkfs -t ext4 /dev/mapper/storage-sambashare
  
    # new mount dir
    sudo mkdir /mnt/sambashare
    sudo chown -R andre:andre /mnt/sambashare
  
    # mount or add to /etc/fstab
    sudo mount /dev/mapper/storage-sambashare /mnt/sambashare
  

• install the required server apps and edit configurations

    sudo apt install samba -y
  
    # edit config
    sudo vim /etc/samba/smb.conf
    # optionally comment out printers under [printers] and [print$]
    # add the lines below
    #[sambashare]
    #  comment = Samba on Ubuntu
    #  path = /mnt/sambashare
    #  read only = no
    #  browsable = yes
  
    # start smb service
    sudo systemctl start smbd.service
  

• now we can connect via Windows by providing the IP address of our Samba server to the file explorer

• To mount samba via Linux as follows:

    sudo mount -t cifs -o user=andre //192.168.1.10/sambashare /mnt/test
  

Conclusion

We have set up a mini pc capable of sharing storage through NFS and Samba. We can then access these storages via other Linux system via the NFS clients or on Windows samba share client.

Now let's try to create our own private image repository. The Gitea application we set up before has a container registry that is compliant with the Open Container Initiative. As such, we can push and pull docker images to our Gitea.

Prerequisites: we need to have SSL enabled on our Gitea application, check here to find out more on how to enable it.

Pushing to a private repository

• My main PC is using Docker Desktop with WSL2. To push our image successfully, we must enable insecure-registries in the Docker desktop configuration.

  Under Settings -> Docker Engine, add the following:

    {
      "builder": {
        "gc": {
          "defaultKeepStorage": "20GB",
          "enabled": true
        }
      },
      "experimental": false,
      "insecure-registries": [
        "192.168.1.4:3000"
      ]
    }
  

  192.168.1.4:3000 is the URL of our Gitea application. This can be a simple domain name such as gitea.local if the DNS record is added into the DNS server (e.g. on PiHole).

• Perform a login to save credentials, and enter the username and password used for Gitea login. Here I have an account name called andre.

    docker login 192.168.1.4:3000 --username andre --password *******
  

• Next, we need to tag the docker image before pushing. According to Gitea documentation:

  

  We need to first tag our image according to the above format

    # tag with an image ID
    docker tag 9b0f43d3c161 192.168.1.4:3000/andre/api-server:1.0.1
  
    # tag when building image
    docker build -t 192.168.1.4:3000/andre/api-server:1.0.1 .
  

• Now we can push to Gitea as shown below

    # push to gitea
    docker push 192.168.1.4:3000/andre/api-server:1.0.1
  
    # to view all the list of images you have pushed
    curl -ik --user andre https://192.168.1.4:3000/v2/_catalog
    # output
    # {"repositories":["andre/api-server"]}
  

    # to view all the tags of image
    curl -ik --user andre https://192.168.1.4:3000/v2/andre/api-server/tags/list
    # output
    # {"name":"andre/api-server","tags":["1.0.1"]}
  

• Here is a list of APIs we can use to view the images we have pushed

Pulling from a private repository

we are following their guide here

we first create our auth value which comprises of <username>:<password> which is then baseb4 encoded. We use -n to prevent echo from adding extra newlines.

echo -n "andre:12345678" | base64
# output
# YW5kcmU6MTIzNDU2Nzg=

Next, we can create our dockerconfig.json to store our credentials secrets. The "192.168.1.4:3000" key is the IP and port where our image repository is located, we provide the credential info such as username, password, email and auth, which we had just encoded earlier into this JSON file.

// dockerconfig.json
{
    "auths": {
        "192.168.1.4:3000": {
            "username": "andre",
            "password": "12345678",
            "email": "andre@mail.com",
            "auth": "YW5kcmU6MTIzNDU2Nzg="
        }
    }
}

Now we can parse this JSON file into base64 again.

base64 dockerconfig.json
# output
# eyJhdXRocyI6eyIxOTIuMTY4LjEuNDozMDAwIjp7InVzZXJuYW1lIjoiYW5kcmUiLCJwYXNzd29yZCI6IjEyMzQ1Njc4IiwiZW1haWwiOiJhbmRyZUBtYWlsLmNvbSIsImF1dGgiOiJZVzVrY21VNk1USXpORFUyTnpnPSJ9fX0=

using this output value, we can then create a image-pull-secret.yaml file and store this value under data.\.dockerconfigjson. The type of secret here is kubernetes.io/dockerconfigjson, which is used to authenticate with a container registry when pulling a private image.

# image-pull-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name:  image-pull-secret
data:
  .dockerconfigjson:  eyJhdXRocyI6eyIxOTIuMTY4LjEuNDozMDAwIjp7InVzZXJuYW1lIjoiYW5kcmUiLCJwYXNzd29yZCI6IjEyMzQ1Njc4IiwiZW1haWwiOiJhbmRyZUBtYWlsLmNvbSIsImF1dGgiOiJZVzVrY21VNk1USXpORFUyTnpnPSJ9fX0= # dockerconfigjson is a base64 encoded string:  cat ~/.docker/config.json | base64 -w 0 
type: kubernetes.io/dockerconfigjson

Now we just need to include this secret in our api-server.yaml so that it knows to use this to authenticate our Gitea application

We can add imagePullSecrets it under the deployment spec.template.spec.

Change the spec.template.spec.containers.image to the one pointing to your private image repository as well

image: 192.168.1.4:3000/andre/api-server:1.0.1

# api-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server-deployment
  annotations:
    description: "this is my normal backend server deployment"
    version: "1.0"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api-server-container
          image: 192.168.1.4:3000/andre/api-server:1.0.1 # update the link to image registry
          resources: # declare resources limits and request
            limits:
              memory: "512Mi"
              cpu: "1"
            requests:
              memory: "256Mi"
              cpu: "0.2"
          ports: # container ports exposed
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: homek8-configmap
            - secretRef:
                name: homek8-secret
      imagePullSecrets:
        - name: image-pull-secret

Making deployments to k8s

Now we are ready to deploy to Kubernetes.

First, we need to ensure that we added our Gitea SSL certificate to our Kubernetes nodes. To do that, copy the giteaCA.crt over to our k8 nodes then update the ca-certificate:

sudo cp giteaCA.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

Next, we can create a namespace for deployment:

kubectl create namespace homek8-custom-registry
kubectl config set-context --current --namespace=homek8-custom-registry

Lastly, deploy

kubectl apply -f .

Now we can see the container being successfully pulled and created.

1. have a running Kubernetes cluster. Check out here to create one yourself

2. have a copy of our backend server code and image. Check out here to create

We will deploy 1 backend-server image, 1 MongoDB image, 1 Redis image, and 1 InfluxDB image to our Kubernetes cluster.

Writing our Yaml files

Let's first set our 3 databases.

1. Mongo

    # mongo.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: mongo-deployment
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: mongo
      template:
        metadata:
          labels:
            app: mongo
        spec:
          containers:
          - name: mongo
            image: mongo:4.4.18
            resources:
              limits:
                memory: "128Mi"
                cpu: "500m"
            ports:
            - containerPort: 27017
    ---
   

   Let's go through what we had written.

   • apiVersion is specified as apps/v1, this is the API version of the Kubernetes object being deployed

   • kind is Deployment

   • metadata.name is mongo-deployment. This is the name of our deployment. Metadata can contain descriptions of our deployments

   • spec describes the state of our deployment.

     • replicas sets how many instances of pods we should be running, sets to 1 here

     • selector.matchLabels specifies which pods will be managed by the deployment based on the labels. This value here must match the pod label that we will be creating. Here we set to match the label mongo

     • template defines the pod used by the deployment to create new pods

       • metadata.labels is used to label our pod, which we had labeled as mongo, this will be referenced by our deployment and service resource

       • spec specify details of our containers, such as the name, image, resources and ports

         • name specifies the name of our container

         • ports specifies the ports that the container exposes

           we want to expose 27017 to other services so we define ports as such:

             ports:
             - containerPort: 27017
           

         • resources specifies the requests and limits for the container. we want to limit the CPU and memory of our container, as such we define our resources as such:

             resources:
               limits:
                 memory: "128Mi"
                 cpu: "500m"
           

         • image specifies the docker image to use for the container

           We will be using the docker image from the docker hub and the image can be defined as <image_name>:<tag>, we are using version 4.4.18 here. (Mongo version 5+ might not run on Kubernetes so we are using an older version, check out this issue)

Next, we can define the service spec

    # mongo.yaml
    # deployment written earlier
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name:  mongo-service
    spec:
      selector:
        app:  mongo
      ports:
      - protocol: TCP
        port:  27017
        targetPort:  27017

Let's go through what we had written.

• apiVersion specifies the API version for the service

• kind is a Service, a Service is a Kubernetes resource that allows a way for pods to be accessed.

• metadata.name specifies the name of our service as mongo-service

• spec describes the state of our service

  • selector specifies the set of pods that follow this service configuration, mainly through the labels. We set this value to be the same label set earlier, mongo

  • ports specifies the ports exposed by the service

    • protocol is the network protocol used, we use tcp here

    • port is the port number we want to be exposed to, 27017

    • targetPort specifies the port on the pod targeted by the service

    • The service maintains its range of ports used (port) and this is mapped to the port used on the pod (targetPort)

2. Redis

   This is similar to Mongo deployment, the only difference is Redis port we will use 6379

    # redis.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: redis-deployment
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: redis
      template:
        metadata:
          labels:
            app: redis
        spec:
          containers:
          - name: redis
            image: redis:alpine
            resources:
              limits:
                memory: "128Mi"
                cpu: "500m"
            ports:
            - containerPort: 6379
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name:  redis-service
    spec:
      selector:
        app:  redis
      ports:
      - protocol: TCP
        port:  6379
        targetPort:  6379
   

3. InfluxDB

   For InfluxDB, other than changing the port to 8086, it also has a few environmental variables that need to be set for the containers. InfluxDB also exposes a web UI that we can access so we will need to expose this to the network outside our Kubernetes.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: influx-deployment
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: influx
      template:
        metadata:
          labels:
            app: influx
        spec:
          containers:
          - name: influx
            image: influxdb:alpine
            resources:
              limits:
                memory: "128Mi"
                cpu: "500m"
            ports:
            - containerPort: 8086
            env:
              - name: DOCKER_INFLUXDB_INIT_MODE
                value: "setup"
              - name: DOCKER_INFLUXDB_INIT_USERNAME
                value: "andre"
              - name: DOCKER_INFLUXDB_INIT_PASSWORD
                value: "12345678"
              - name: DOCKER_INFLUXDB_INIT_ORG
                value: "andre"
              - name: DOCKER_INFLUXDB_INIT_BUCKET
                value: "bucket1"
              - name: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
                value: "my-token"
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name:  influx-service
    spec:
      type: NodePort
      selector:
        app:  influx
      ports:
      - protocol: TCP
        port:  8086
        targetPort:  8086
        nodePort: 32000
   

   Explanations:

   • spec.template.spec.env specifies our environmental variables for the containers. We have written a few that are required by the InfluxDB image, for example, the password and the admin token. We will take this out later and put it under Secrets.

   • spec.type under Service is set to NodePort. NodePort here specifies the port number allocated on every node of the cluster. This is used to access the service externally, mainly to access our web UI

4. API server

   Now we can write our deployment for api-server. We will be using port 8080 for our container and port 80 for our service. This maps port 80 of the service to port 8080 of the container.

   We also declared the ENV and NAMESPACE environmental variables for this container to be used in our code later.

   Our image for the api-server is using the one we had previously uploaded to the docker hub, andrewongzh/api-server:1.0.0. This is in the format of <username>/<repo_name>:<tag_name>

    # api-server.yaml
    apiVersion: v1
    kind: Service
    metadata:
     name: api-server-service
    spec:
      type: NodePort
      selector:
        app: api-server
      ports:
        - protocol: TCP
          port: 80 
          targetPort: 8080 # the port number which the service will forward to
          nodePort: 32001 # this port is exposed externally outside of k8s
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: api-server-deployment
      annotations:
        description: "this is my normal backend server deployment"
        version: "1.0"
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: api-server
      template:
        metadata:
          labels:
            app: api-server
        spec:
          containers:
            - name: api-server-container
              image: andrewongzh/api-server:1.0.1 # update the link to image registary
              resources: # declare resources limits and request
                limits:
                  memory: "512Mi"
                  cpu: "1"
                requests:
                  memory: "256Mi"
                  cpu: "0.2"
              ports: # container ports exposed
                - containerPort: 8080
              env:
                - name: TEST
                  value: "1-2-3"
                - name: ENV
                  value: "k8"
                - name: NAMESPACE
                  value: "homek8"
                - name: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
                  value: "my-token"
   

Updating API-server code

Using the same ENV key we defined before, we can add an if else statement to check ENV == "k8" for Kubernetes deployment. If so, we can then change the URI of our client DB connection and replace it with the format of <URI_scheme>://<service-name>.<namespace>.svc.cluster.local:<port>. service-name is what we defined in our Service yaml earlier, which will resolve to the IP address via internal DNS resolution. namespace is the namespace we decided to deploy on which is homek8.

var ENV = os.Getenv("ENV")
var nameSpace = os.Getenv("NAMESPACE")

func MongoClientInit() *mongo.Client {
    defer cancel()
    address := "mongodb://localhost:27017"
    if ENV == "dockercompose" {
        address = "mongodb://mongo:27017"
    } else if ENV == "k8" {
        // hard coded service name here
        address = fmt.Sprintf("mongodb://mongo-service.%s.svc.cluster.local:27017", nameSpace)
    }
    mongoClient, err := mongo.Connect(ctx, options.Client().ApplyURI(address))
    if err != nil {
        panic(err)
    }
    if err := mongoClient.Ping(ctx, nil); err != nil {
        panic(err)
    }

    fmt.Println("Connect to mongodb OK")
    return mongoClient
}

func RedisClientInit() *redis.Client {
    address := "localhost:6379"
    if ENV == "dockercompose" {
        address = "redis:6379"
    } else if ENV == "k8" {
        // hard coded service name here
        address = fmt.Sprintf("redis-service.%s.svc.cluster.local:6379", nameSpace)
    }
    rdb := redis.NewClient(&redis.Options{
        Addr:     address,
        Password: "",
        DB:       0,
    })

    if err := rdb.Ping(redisCtx); err.Err() != nil {
        panic(err)
    }

    fmt.Println("Connect to redis OK")
    return rdb
}

func InfluxClientInit() *influxdb2.Client {
    address := "http://localhost:8086"
    if ENV == "dockercompose" {
        address = "http://influx:8086"
    } else if ENV == "k8" {
        // hard coded service name here
        address = fmt.Sprintf("http://influx-service.%s.svc.cluster.local:8086", nameSpace)
    }
    ifxdb := influxdb2.NewClient(address, influxToken)
    ok, err := ifxdb.Ping(influxCtx)
    if err != nil {
        panic(err)
    }
    if !ok {
        panic("Unable to connect to influxdb")
    }
    fmt.Println("Connect to influx OK")
    return &ifxdb
}

Let's rebuild the image and push it to docker hub tagged as 1.0.1

docker build . -t andrewongzh/api-server:1.0.1
docker push andrewongzh/api-server:1.0.1

Deploying to Kubernetes

We will ssh into our machine and use the kubectl command to execute our deployments.

First, we can create a namespace called homek8 for this deployment. A namespace allows us to separate our deployments from the rest of the deployments in Kubernetes clusters. This is useful if we want to isolate resources via different users or projects.

kubectl create namespace homek8

Next, we can start deploying

# -f referes to the yaml file we want to use
kubectl apply -f influx.yaml --namespace=homek8
kubectl apply -f mongo.yaml -n homek8
kubectl apply -f redis.yaml -n homek8
kubectl apply -f api-server.yaml -n homek8

We can verify if our deployments are successful

kubectl get pods -n homek8
# output
NAME                                 READY   STATUS    RESTARTS   AGE
influx-deployment-5c875ffdbc-rmh9w   1/1     Running   0          28s
mongo-deployment-67b6bbf7fd-66thz    1/1     Running   0          10s
redis-deployment-f76b8d78c-n7r4d     1/1     Running   0          5s

When we create a deployment, it creates a replica set and names it as <deployment_name>-<random-string>. In the case of our influxDB deployment, the replica set name is influx-deployment-5c875ffdbc. The replica set then creates the pods which will add another random string to the back which is rmh9w.

We can also try to access our influxDB web UI via http://192.168.1.95:32000/, where the IP is any of our nodes and the port is previously defined under nodePort , which is 32000.

To test our API server, we can make a simple curl command to it

curl http://192.168.1.95:32001
# output
{"hello":"world"}

Let's change our current context to always use the home8 namespace so we don't have to keep typing it

kubectl config set-context --current --namespace=homek8

Let's look at our api-server's log, we can either dump out the logs or stream it

# get logs via pod name
# -f to stream the logs
kubectl logs -f api-server-deployment-77fb9b8bcc-sx6c8

# get logs via deployment name
kubectl logs deploy/api-server-deployment

Let's increase the number of replicas for our api-server to 2 then apply our yaml file again.

kubectl apply -f api-server.yaml
kubectl get pods
# now we get 2 pod names
api-server-deployment-77fb9b8bcc-sx6c8   1/1     Running   0          16m
api-server-deployment-77fb9b8bcc-xnbnc   1/1     Running   0          104s

If we want to delete our deployments, we can use kubectl delete

kubectl delete -f influx.yaml
kubectl delete -f mongo.yaml
kubectl delete -f redis.yaml

Managing environmental variables

Let's move the environmental variables into ConfigMap and Secret.

# config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: homek8-configmap
data:
  TEST: "1-2-3"
  ENV: "k8"
  NAMESPACE: "homek8"
  DOCKER_INFLUXDB_INIT_MODE: "setup"
  DOCKER_INFLUXDB_INIT_USERNAME: "andre"
  DOCKER_INFLUXDB_INIT_ORG: "andre"
  DOCKER_INFLUXDB_INIT_BUCKET: "bucket1"

We can convert our password into baseb4 encoded required in the secret.yaml via echo -n "12345678" | baseb4

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name:  homek8-secret
type: Opaque
data:
  DOCKER_INFLUXDB_INIT_ADMIN_TOKEN:  bXktdG9rZW4= # base64 encoded of my-token
  DOCKER_INFLUXDB_INIT_PASSWORD: MTIzNDU2Nzg= # base64 encoded of 12345678

In our deployment, we can reference it in 2 different ways:

Use envFrom to reference the name of our configMap or secret via configMapRef and secretRef and reference our name homek8-configmap and homek8-secret

The second way is to use valueFrom.secretKeyRef or valueFrom.configMapKeyRef to reference the name and key of our resources under env.

# influx.yaml
    spec:
      containers:
      - name: influx
        image: influxdb:alpine
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8086
        envFrom:
        - configMapRef:
            name: homek8-configmap
        env:
        - name: DOCKER_INFLUXDB_INIT_PASSWORD
          valueFrom:
            secretKeyRef:
              name:  homek8-secret
              key: DOCKER_INFLUXDB_INIT_PASSWORD
        - name: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
          valueFrom:
            secretKeyRef:
              name: homek8-secret
              key: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN

# api-server.yaml
envFrom:
- configMapRef:
    name: homek8-configmap
- secretRef:
    name: homek8-secret

Manage volumes for our databases

We can add persistent volume for our instances so that our data will not disappear if the database instances have crashed.

We can create our Persistent Volume for our claims later.

# pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongo-pv
  labels:
    name: mongo-pv
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /mongo
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: influx-pv
  labels:
    name: influx-pv
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /influx
---

metadata.labels specifies the label on our PV, which will be referenced by the PVC later.

accessModes specifies how many pods can access and how many nodes can mount this volume

resources.requests.storage specifies the size of the storage we require

hostPath specifies the location of where this storage mounts a folder on the node's filesystem (not recommended by k8 but an easy way for testing)

We can create a Persistent Volume Claim for each of our DB which is a storage request.

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongo-pvc
spec: 
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      name: mongo-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: influx-pvc
spec: 
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      name: influx-pv
---

metadata.name here can be referenced later in our deployment

accessModes specifies how many pods can access and how many nodes can mount this volume

resources.requests.storage specifies the size of the storage we require

selector.matchLabels specifies the persistent volume to bind to

We can then apply this to our Kubernetes cluster: kubectl apply -f pvc.yaml -f pv.yaml and we can view their status as below

kubectl get pvc
# output
NAME         STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
influx-pvc   Bound    influx-pv   1Gi        RWO                           6m54s
mongo-pvc    Bound    mongo-pv    1Gi        RWO                           6m54s

kubectl get pv
# output
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
influx-pv   1Gi        RWO            Retain           Bound    homek8/influx-pvc                           6m52s
mongo-pv    1Gi        RWO            Retain           Bound    homek8/mongo-pvc                            6m52s

Now we can update our DB deployments to include volumes:

# mongo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - name: mongo
        image: mongo:4.4.18
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 27017
        volumeMounts:
          - name:  mongo-storage # name of the volume
            mountPath:  /data/db
      volumes:
      - name: mongo-storage
        persistentVolumeClaim:
          claimName: mongo-pvc
---

For Mongo, we are mounting it to /data/db

For InfluxDB, we are mounting it to /var/lib/influxdb

Conclusion

We learned how to create deployment yaml files and deploy them to our Kubernetes clusters. We used a few Kubernetes api-resources such as Deployment, Service, ConfigMap, Secret, PersistentVolume, and PersistentVolumeClaims.

We will be creating our own backend API server that we can use to test out our Kubernetes deployment. We will also be running this on a docker-compose setup for testing as well. We will connect to Mongo Database, Redis, and Influx Database to store and query our data.

API endpoint that we will create:

path "/", is just a GET method to test if we can reach our API-server, it will just return a simple JSON object {"hello": "world"}

path "/note", a POST method for users to insert their notes into the database, and the GET method to retrieve their desired notes.

Prerequisites

Some things to installed beforehand:

1. Golang

2. Docker engine and Docker Compose

Writing API server code

1. initialize go mod file and install necessary dependencies

    mkdir api-server
    cd api-server
    go mod init api-server
   
    # install dependencies
    go get github.com/gin-gonic/gin
    go get go.mongodb.org/mongo-driver/mongo
    go get github.com/go-redis/redis/v9
    go get github.com/influxdata/influxdb-client-go/v2
   

2. write a simple gin router to start a server in main.go

    // main.go
   
    package main
    import (
        ...
    )
   
    func main() {
        fmt.Println("homek8")
        }()
   
        r := gin.Default()
   
        // simple get function to test
        r.GET("/", func(c *gin.Context) {
            c.JSON(http.StatusOK, gin.H{
                "hello": "world",
            })
        })
   
        r.Run()
    }
   

3. write the clients for Mongo, Redis, and Influxdb to connect to the database.

   Once done we can add the connection to our main function. Remember to close the connection for the clients by adding it to the defer statement.

    // main.go
   
    var influxToken = os.Getenv("DOCKER_INFLUXDB_INIT_ADMIN_TOKEN")
   
    var ctx, cancel = context.WithTimeout(context.Background(), 10*time.Second)
    var redisCtx = context.Background()
    var influxCtx = context.Background()
   
    func MongoClientInit() *mongo.Client {
        defer cancel()
        address := "mongodb://localhost:27017"
        mongoClient, err := mongo.Connect(ctx, options.Client().ApplyURI(address))
        if err != nil {
            panic(err)
        }
        if err := mongoClient.Ping(ctx, nil); err != nil {
            panic(err)
        }
   
        fmt.Println("Connect to mongodb OK")
        return mongoClient
    }
   
    func RedisClientInit() *redis.Client {
        address := "localhost:6379"
        rdb := redis.NewClient(&redis.Options{
            Addr:     address,
            Password: "",
            DB:       0,
        })
   
        if err := rdb.Ping(redisCtx); err.Err() != nil {
            panic(err)
        }
   
        fmt.Println("Connect to redis OK")
        return rdb
    }
   
    func InfluxClientInit() *influxdb2.Client {
        address := "http://localhost:8086"
        ifxdb := influxdb2.NewClient(address, influxToken)
        ok, err := ifxdb.Ping(influxCtx)
        if err != nil {
            panic(err)
        }
        if !ok {
            panic("Unable to connect to influxdb")
        }
        fmt.Println("Connect to influx OK")
        return &ifxdb
    }
   
    func main() {
        fmt.Println("homek8")
        fmt.Println("connecting to clients ...")
        redisClient := RedisClientInit()
        mongoClient := MongoClientInit()
        influxClient := InfluxClientInit()
        fmt.Println("connected")
   
        defer func() {
            if err := mongoClient.Disconnect(ctx); err != nil {
                panic(err)
            }
            (*influxClient).Close()
        }()
   
        ...
    }
   

4. write our API handler for /note

   The POST handler /note will write the title and value information into our Mongo database.

   The GET handler /note will first try to retrieve the data from the Redis database. If that fails, it will query the Mongo database and update the entry into the Redis database.

    // main.go
   
    type Note struct {
        Title string `json:"title"`
        Value string `json:"value"`
    }
   
    func main() {
        ...initiallization...
        coll := mongoClient.Database("homek8").Collection("notes")
   
        r.GET("/note", func(c *gin.Context) {
            name, _ := c.Params.Get("name")
   
            val, err := redisClient.Get(redisCtx, name).Result()
            if err != nil && err != redis.Nil {
                log.Println(err)
                c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get note"})
                return
            }
            if err == redis.Nil {
                fmt.Println("found redis value:", val)
                c.JSON(http.StatusOK, Note{Title: name, Value: val})
                return
            }
   
            var note Note
            filter := bson.D{{Key: "name", Value: name}}
            err = coll.FindOne(ctx, filter).Decode(&note)
            if err != nil {
                c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get note"})
                return
            }
   
            err = redisClient.Set(redisCtx, name, note.Value, 0).Err()
            if err != nil {
                fmt.Println("failed to cache data into redis")
            }
            c.JSON(http.StatusOK, note)
        })
   
        r.POST("/note", func(c *gin.Context) {
            var note Note
            err := c.BindJSON(&note)
            if err != nil {
                fmt.Println("unmarshall: ", err)
                c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to add a note"})
                return
            }
   
            res, err := coll.InsertOne(ctx, note)
            if err != nil {
                fmt.Println("mongo error: ", err)
                c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to add a note"})
                return
            }
            id := res.InsertedID
            c.JSON(http.StatusOK, gin.H{"success": id})
        })
   
        ...
    }
   

5. write a go routine to send data to influxdb

   we will send random average and max values to our influxdb database. It will emit one value once every 5 seconds for 100 times

    func main() {
        writeAPI := (*influxClient).WriteAPIBlocking("andre", "bucket1")
        go func() {
            count := 100
            avg := 24.5
            max := 45.0
            for count > 0 {
                avgDeviation := rand.Intn(20) + 1
                maxDeviation := rand.Intn(6) + 0
                pt := influxdb2.NewPoint("stat",
                    map[string]string{"unit": "temperature"},
                    map[string]interface{}{"avg": avg + float64(avgDeviation), "max": max + float64(maxDeviation)},
                    time.Now())
                err := writeAPI.WritePoint(influxCtx, pt)
                if err != nil {
                    fmt.Println("emit point error", err)
                } else {
                    fmt.Println("success sent to influx db")
                }
                time.Sleep(time.Second * 5)
                count -= 1
            }
        }()
    }
   

6. run go build to check if there are no errors

Create a Dockerfile

Let's create a docker file for us to containerize our application by creating a Dockerfile

# image will be build using Go version 1.21
FROM golang:1.21

# sets the current working directory to /app
WORKDIR /app
# copy our go.mod and go.sum into our current working directory
COPY go.mod go.sum ./
# install our go dependencies
RUN go mod download
# copy our main.go (or all go files) to our current working directory
COPY *.go ./
# build our application and name our binary file as output
RUN go build -o output

# expose 8080 in our container
EXPOSE 8080

# execut command to launch our appliaction
CMD [ "/app/output" ]

Here, we can manually build our image using this Dockerfile by running: docker build . .It will search for any Dockerfile in the current directory and build the image.

We can also tag our images when building with a -t <tag>

To view all created images, run docker images

Create a docker-compose file

Now we can write a docker-composee.yml file to spin up the containers we need. We need 4 containers, our backend server we had just written, a mongo db container, a redis db container, and an influx db container. We can easily spin up multiple docker containers with a single command with docker-compose.

services:
  backend:
    build: .
    ports:
      - "8080:8080"
    environment:
      - ENV=dockercompose
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-token
    depends_on:
      - mongo
      - redis
      - influx
  mongo:
    image: mongo:6.0
    ports:
      - "27017:27017"
  redis:
    image: redis:alpine
    ports:
      - "6379:6379"
  influx:
    image: influxdb:alpine
    environment:
      - DOCKER_INFLUXDB_INIT_USERNAME=andre
      - DOCKER_INFLUXDB_INIT_PASSWORD=12345678
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_ORG=andre
      - DOCKER_INFLUXDB_INIT_BUCKET=bucket1
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-token
    ports:
      - "8086:8086"

In our docker-compose.yml file, we list out the names of the services that we need: backend, mongo, redis, influx.

Under the backend, we have build . which will allow Docker Compose to look for the Dockerfile we had written earlier and use that to build the image for this service. For the other services, we will be using images from Docker Hub, for example, mongo:6.0 , redis:alpine.

The ports we defined above also expose the ports of each container to our host.

We also added a new environment variable ENV=dockercompose to our backend container. This allows us to change the URI of our databases easily. We also pass in our influx admin token to the backend so it can successfully emit data to the influx database.

depends_on over here will wait for Mongo, redis, and influx services to be up and running first before running the backend service.

Now we need to update our backend code to include this ENV variable. We can get the ENV environment variable through the code and use it to set the client address URL we will connect to. We replace localhost here to the name of our service which will then automatically resolve to the correct IP address.

Under the hood, docker-compose sets up its own DNS resolution that maps the service name to its IP address. This is used for communications between containers.

// main.go

var ENV = os.Getenv("ENV")

func MongoClientInit() *mongo.Client {
    defer cancel()
    address := "mongodb://localhost:27017"
    if ENV == "dockercompose" {
        address = "mongodb://mongo:27017"
    }
    mongoClient, err := mongo.Connect(ctx, options.Client().ApplyURI(address))
    if err != nil {
        panic(err)
    }
    if err := mongoClient.Ping(ctx, nil); err != nil {
        panic(err)
    }

    fmt.Println("Connect to mongodb OK")
    return mongoClient
}

func RedisClientInit() *redis.Client {
    address := "localhost:6379"
    if ENV == "dockercompose" {
        address = "redis:6379"
    }
    rdb := redis.NewClient(&redis.Options{
        Addr:     address,
        Password: "",
        DB:       0,
    })

    if err := rdb.Ping(redisCtx); err.Err() != nil {
        panic(err)
    }

    fmt.Println("Connect to redis OK")
    return rdb
}

func InfluxClientInit() *influxdb2.Client {
    address := "http://localhost:8086"
    if ENV == "dockercompose" {
        address = "http://influx:8086"
    }
    ifxdb := influxdb2.NewClient(address, influxToken)
    ok, err := ifxdb.Ping(influxCtx)
    if err != nil {
        panic(err)
    }
    if !ok {
        panic("Unable to connect to influxdb")
    }
    fmt.Println("Connect to influx OK")
    return &ifxdb
}

Verify if it's working

Now we can run containers using docker-compose: docker compose up. We should be able to see 4 containers running.

To stop running our containers : docker compose down

Docker Image

Now we can push our image to our image repository. Here we will be pushing to Docker hub. After creating an account on the website, we can create a new repository as shown below.

We can then build our image with the command: docker build . -t andrewongzh/api-server:1.0.0

we might need to login to docker hub first in the terminal

run docker login -u <username>

After that, we can push our image with the command: docker push andrewongzh/api-server:1.0.0. This is in the format of <username>/<image name>:<image tag>

one thing to note here is to tag the image name correctly to match the format of how we are pushing to the repository

If everything works well, we can see our image on Docker hub.

In conclusion, we created a simple backend server to play around with docker and docker-compose. We will use this server for more things in the future.

Handle secrets

Some extra we can do is to move our secrets into a file instead of writing it on the docker-compose.yml file.

First, we can create a folder to store our secrets and then create influxtoken.txt and passwd.txt under it.

mkdir secrets
echo my-token > ./secrets/influxtoken.txt
echo 12345678 > ./secrets/passwd.txt

In the docker-compose.yml file make the following changes:

services:
  backend:
    build: .
    ports:
      - "8080:8080"
    environment:
      - ENV=dockercompose
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=/run/secrets/influxtoken
    depends_on:
      - mongo
      - redis
      - influx
    secrets:
      - passwd
  mongo:
    image: mongo:6.0
    ports:
      - "27017:27017"
  redis:
    image: redis:alpine
    ports:
      - "6379:6379"
  influx:
    image: influxdb:alpine
    environment:
      - DOCKER_INFLUXDB_INIT_USERNAME=andre
      - DOCKER_INFLUXDB_INIT_PASSWORD=/run/secrets/passwd
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_ORG=andre
      - DOCKER_INFLUXDB_INIT_BUCKET=bucket1
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=/run/secrets/influxtoken
    ports:
      - "8086:8086"
secrets:
  passwd:
    file: ./secrets/passwd.txt
  influxtoken:
    file: ./secrets/influxtoken.txt

Appendix

// full code of main.go

package main

import (
    "context"
    "fmt"
    "log"
    "math/rand"
    "net/http"
    "os"
    "time"

    "github.com/gin-gonic/gin"
    influxdb2 "github.com/influxdata/influxdb-client-go/v2"
    "github.com/redis/go-redis/v9"
    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

type Note struct {
    Title string `json:"title"`
    Value string `json:"value"`
}

var ENV = os.Getenv("ENV")
var influxToken = os.Getenv("DOCKER_INFLUXDB_INIT_ADMIN_TOKEN")

var ctx, cancel = context.WithTimeout(context.Background(), 10*time.Second)
var redisCtx = context.Background()
var influxCtx = context.Background()

func MongoClientInit() *mongo.Client {
    defer cancel()
    address := "mongodb://localhost:27017"
    if ENV == "dockercompose" {
        address = "mongodb://mongo:27017"
    }
    mongoClient, err := mongo.Connect(ctx, options.Client().ApplyURI(address))
    if err != nil {
        panic(err)
    }
    if err := mongoClient.Ping(ctx, nil); err != nil {
        panic(err)
    }

    fmt.Println("Connect to mongodb OK")
    return mongoClient
}

func RedisClientInit() *redis.Client {
    address := "localhost:6379"
    if ENV == "dockercompose" {
        address = "redis:6379"
    }
    rdb := redis.NewClient(&redis.Options{
        Addr:     address,
        Password: "",
        DB:       0,
    })

    if err := rdb.Ping(redisCtx); err.Err() != nil {
        panic(err)
    }

    fmt.Println("Connect to redis OK")
    return rdb
}

func InfluxClientInit() *influxdb2.Client {
    address := "http://localhost:8086"
    if ENV == "dockercompose" {
        address = "http://influx:8086"
    }
    ifxdb := influxdb2.NewClient(address, influxToken)
    ok, err := ifxdb.Ping(influxCtx)
    if err != nil {
        panic(err)
    }
    if !ok {
        panic("Unable to connect to influxdb")
    }
    fmt.Println("Connect to influx OK")
    return &ifxdb
}

func main() {
    fmt.Println("homek8")
    fmt.Println("connecting to clients ...")
    redisClient := RedisClientInit()
    mongoClient := MongoClientInit()
    influxClient := InfluxClientInit()
    fmt.Println("connected")

    defer func() {
        if err := mongoClient.Disconnect(ctx); err != nil {
            panic(err)
        }
        (*influxClient).Close()
    }()
    coll := mongoClient.Database("homek8").Collection("notes")
    writeAPI := (*influxClient).WriteAPIBlocking("andre", "bucket1")

    // we shall emit data every 5 seconds for 100 times
    go func() {
        count := 100
        avg := 24.5
        max := 45.0
        for count > 0 {
            avgDeviation := rand.Intn(20) + 1
            maxDeviation := rand.Intn(6) + 0
            pt := influxdb2.NewPoint("stat",
                map[string]string{"unit": "temperature"},
                map[string]interface{}{"avg": avg + float64(avgDeviation), "max": max + float64(maxDeviation)},
                time.Now())
            err := writeAPI.WritePoint(influxCtx, pt)
            if err != nil {
                fmt.Println("emit point error", err)
            } else {
                fmt.Println("success sent to influx db")
            }
            time.Sleep(time.Second * 5)
            count -= 1
        }
    }()

    r := gin.Default()

    // simple get function to test
    r.GET("/", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "hello": "world",
        })
    })

    r.GET("/note", func(c *gin.Context) {
        name, _ := c.Params.Get("name")

        val, err := redisClient.Get(redisCtx, name).Result()
        if err != nil && err != redis.Nil {
            log.Println(err)
            c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get note"})
            return
        }
        if err == redis.Nil {
            fmt.Println("found redis value:", val)
            c.JSON(http.StatusOK, Note{Title: name, Value: val})
            return
        }

        var note Note
        filter := bson.D{{Key: "name", Value: name}}
        err = coll.FindOne(ctx, filter).Decode(&note)
        if err != nil {
            c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get note"})
            return
        }

        err = redisClient.Set(redisCtx, name, note.Value, 0).Err()
        if err != nil {
            fmt.Println("failed to cache data into redis")
        }
        c.JSON(http.StatusOK, note)
    })

    r.POST("/note", func(c *gin.Context) {
        var note Note
        err := c.BindJSON(&note)
        if err != nil {
            fmt.Println("unmarshall: ", err)
            c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to add a note"})
            return
        }

        res, err := coll.InsertOne(ctx, note)
        if err != nil {
            fmt.Println("mongo error: ", err)
            c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to add a note"})
            return
        }
        id := res.InsertedID
        c.JSON(http.StatusOK, gin.H{"success": id})
    })

    r.Run()
}

Secure Socket Layer is a protocol that provides secure communication over two network devices. It ensures that the data transmitted between two devices remains private. The most common example would be the client computer (us) connecting to a secure server (for example a bank website).

The server usually has its own private key and public signed certificate. The private key is only known by the server and no one else. Here are the steps to establish SSL communication:

1. The publicly signed certificate is sent to the client for verification with the Certificate Authority (CA)

2. Once verified, the client sends a secret message encoded with the public certificate and sends it back to the server

3. The secret message can be decoded by the private key held by the server and read the contents of the message, thereby creating a new session whereby now only the client and server know this message.

4. Now both the client and server can use this message to encrypt and decrypt data with the session key

Setting up SSL

We will be setting up SSL and HTTPS on our Gitea application that we set up previously. Previously our Gitea application only used HTTP for communication.

We will be exploring two ways to set up SSL/HTTPS:

1. Let Gitea handle SSL encryption/decryption by providing the private and public keys to our application

2. Let our reverse proxy (Nginx) handle SSL encryption/decryption instead and communicate using HTTP between the reverse proxy and Gitea

Option 1

Most of the time applications have the option to enable SSL/HTTPS by specifying the private key and public self-signed certificate. Looking at the Gitea documentation to set up SSL, we need to change a few configurations in the app.ini file.

1. Let's generate a private key for the CA

    # gitea GITEA_CUSTOM path
    cd /data/gitea/
   
    mkdir certs && cd ./certs
   
    # generate private key
    openssl genrsa -out giteaCA.key 2048
   

2. Create a self-signed certificate for the CA

   First, create a csr config to set our required information. The important parts here are the subjectAlternativeName. We will need to add the IP address to the Subject Alternative Name (SAN). This is important later when we want to use Gitea as our Kubernetes image repository.

    [req]
    default_bits = 2048
    prompt = no
    default_md = sha256
    distinguished_name = dn
    req_extensions = req_ext
    default_days = 900
   
    [dn]
    # The Common Name should be the fully qualified domain name (FQDN) of the server.
    # Replace "example.com" with your domain name.
    CN = 192.168.1.4
   
    [req_ext]
    # Subject Alternative Name (SAN) extension
    subjectAltName = @alt_names
   
    [alt_names]
    # Add additional subject alternative names (SANs) here.
    IP = 192.168.1.4
   

   After that we can use this csr to generate our certificate

3.   openssl req -new -x509 -key giteaCA.key -out giteaCA.crt -config csr -extensions req_ext
   

   `-x509` : output should be an X.509 certificate instead of a certificate signing request

   -days : how long this certificate is valid for

   -key : the private key file used for generating the public certificate

   -out : specifies the name of the newly created public certificate

   -config : specifies the location of our configuration file

   -externsions : specifies that we are using extensions by passing in the var req_ext in our config file

4. Now we have giteaCA.key and giteaCA.crt

5. Looking at the Gitea app.ini file here. Our ROOT_URL here should be using HTTPS, PROTOCOL set to HTTPS and also we need to provide the path to our CERT_FILE and KEY_FILE as shown below

    [server]
    ...
    HTTP_PORT = 3000
    ROOT_URL = https://192.168.1.4:3000/
    PROTOCOL = https
    CERT_FILE = ./certs/giteaCA.crt
    KEY_FILE = ./certs/giteaCA.key
   

6. After editing the app.ini file we can restart our Gitea application

7. Verify that we can access the application via https://192.168.1.4:3000 , however, it will give us a warning when we are accessing via a browser

   

   This is because the client PC that we are accessing our application failed to validate the public certificate that the server has sent us. To solve this issue, we need to add our giteaCA.crt file to the trusted root certificates. This can be done in different ways on different machines (windows, Mac, Linux, etc...) but won't be covered here for simplicity.

Option 1 is a simple way to set out SSL and HTTPS on the application itself.

Option 2

We will add SSL and HTTPS to our Nginx reverse proxy instead and keep the Gitea application communication using HTTP. Here are a few reasons why we do this:

1. Sometimes enabling SSL on every application can be a hassle. If we have X number of applications needing to enable SSL, then we probably need to generate X private and public keys for each application. This can be reduced to 1 if we only use SSL up to the reverse proxy.

2. We might not want our application to handle SSL as it needs to use extra resources (ie CPU and memory) to handle SSL encryption and decryption.

As such, we sometimes let our network gateway, which is our reverse proxy also be the point where SSL is decrypted, and for the internal network communication is often done through http instead.

1. Let's create our certificate and key again, one-liner to generate it:

    openssl req -new -x509 -days 365 -newkey rsa:2048 -nodes -keyout ca.key -out ca.crt
   

2. Let's update our nginx.conf to include this certificate and key:

    server {
        listen 443 ssl;
        server_name 192.168.1.4;
        ssl_certificate /certs/ca.crt;
        ssl_certificate_key /certs/ca.key;
   
        location /gitea/ {
   
            rewrite ^ $request_uri;
            rewrite ^/gitea(/.*) $1 break;
   
            proxy_pass http://192.168.1.4:3000$uri;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
   

   SSL uses port 443, which we will configure our server to listen to. ssl_certifcate and ssl_certificate_key we will provide a path from our side. We want our client to access Gitea using HTTPS using the domain: https://192.168.1.4/gitea, so we added a /gitea/ location so that it can match our prefix with the URI.

   rewrite ^/gitea(/.*) $1 break; over here will then rewrite our URI from gitea/($1) to ($1), thereby removing the extra gitea/ prefix before calling our proxy_pass directive.

   For example, a client sending a request https://192.168.1.4/gitea/user/login will be modified to http://192.168.1.4/user/login in Nginx before forwarding this request. The URI being rewritten here is /gitea/user/login to /user/login.

3. If we are running our Nginx on a docker container, remember to enable port 443 under the ports section

    version: "1.0"
    services:
      nginx:
        image: nginx:stable
        container_name: nginx
        volumes:
          - /home/internal/nginx/nginx.conf:/etc/nginx/conf.d/nginx.conf
          - /home/internal/nginx/access.log:/var/log/nginx/access.log
          - /home/internal/nginx/error.log:/var/log/nginx/error.log
          - /home/internal/certs/:/certs
        ports:
          - "80:80"
          # for SSL use only
          - "443:443"
   

4. Update the Gitea app.ini file to use the correct domain. This should be 192.168.1.4/gitea

    [server]
    ...
    DOMAIN = 192.168.1.4/gitea
    SSH_DOMAIN = 192.168.1.4/gitea
    ROOT_URL = http://192.168.1.4:3000/gitea
   

Start our Nginx application and try to access our Gitea application through HTTPS again and it should work as well.

What is Kubernetes?

According to their official website, Kubernetes (K8s) is an open-source platform for automating deployment, scaling, and managing containerized applications. Some of K8's features are Automated rollout and rollbacks, availability, resource management, service discovery, and load balancing.

A Kubernetes cluster minimally has two nodes, one being the master and the other being the worker nodes. The master node is also known as the control plane which manages the state of the cluster through etcd, an API server for users to make API calls to query and modify the cluster, schedules newly created pods to worker nodes to run on, and also have various controller manager that watch and handle their specific tasks.

Setting up Kubernetes

We will be using kubeadm to set up our Kubernetes cluster consisting of one master and one worker node. This is a much simpler way to bootstrap our Linux virtual machine with Kubernetes. We will be using Ubuntu-22.04.3-live-server-amd64.iso which can be downloaded from here.

Let's create our first virtual machine and make this our master node. When creating our virtual machine, ensure we have minimally 2 virtual CPU cores and 2GB of RAM. The rest of the settings can be left as default. We will name this node k8-master.

Now start the virtual machine and then proceed with the Ubuntu server installation. We can just go for the defaults.

1. remove the swap from the server, edit the /etc/fstab and comment out the swap, and then reboot the server. We can verify if the swap is disabled by running swapon --show

2. enable IP tables bridged traffic. we need to enable overlay and br_netfilter, as well as to enable bridged traffic to pass through the firewall rules for packet filtering

    echo -e "overlay\nbr_netfilter" | sudo tee /etc/modules-load.d/k8s.conf
   
    sudo modprobe overlay
    sudo modprobe br_netfilter
   
    echo -e "net.bridge.bridge-nf-call-iptables  = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.ipv4.ip_forward                 = 1" | sudo tee /etc/sysctl.d/k8s.conf
   
    # refresh system configurations
    sudo sysctl --system
   

3. Now we can install our container runtime. Here we have some options to choose from, mainly CRI-O, containerd, or Docker Engine. In our setup, we will be using CRI-O. The documentation of how to install can be found here.

   Set up environmental variables and then run the following as root

   1.      OS="xUbuntu_22.04"
           VERSION="1.28"
      
           sudo -s
           echo "deb [signed-by=/usr/share/keyrings/libcontainers-archive-keyring.gpg] https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
           echo "deb [signed-by=/usr/share/keyrings/libcontainers-crio-archive-keyring.gpg] https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
      
           mkdir -p /usr/share/keyrings
           curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | gpg --dearmor -o /usr/share/keyrings/libcontainers-archive-keyring.gpg
           curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/Release.key | gpg --dearmor -o /usr/share/keyrings/libcontainers-crio-archive-keyring.gpg
      
           apt-get update
           apt-get install cri-o cri-o-runc -y
      
           # optional
           apt-get install install cri-tools -y
      
           # enable cri-o
           systemctl daemon-reload
           systemctl enable crio --now
      
           # verify cri-o running
           systemctl status crio
      

4. Install kubeadm, kublet, and kubectl

   We are installing Kubernetes V1.28, documentations can be referred to here

   Run the following commands in root:

    apt-get update
    apt-get install -y apt-transport-https ca-certificates curl gpg
   
    # version number here can be disregarded, they all use the same signing key
    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   
    # add packages for kubernetes 1.28
    echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
   
    apt-get update
    apt-get install -y kubelet kubeadm kubectl
   
    # pin the package version, do not allow them to auto update
    apt-mark hold kubelet kubeadm kubectl
   

Now we can do a clone here to make a copy of our virtual machine so we can replicate it later for our worker nodes.

Bootstrap kubeadm

1. set the following environment variables, ensure that the POD_CIDR does not conflict with the host network's CIDR

   Pod CIDR addresses are the range of network addresses that Kubernetes assigns to pods for communication with each other

    IPADDR="192.168.1.95"
    # nodename will be k8-master
    NODENAME=$(hostname -s)
    POD_CIDR="10.244.0.0/16"
   
    kubeadm init --apiserver-advertise-address=$IPADDR  --apiserver-cert-extra-sans=$IPADDR  --pod-network-cidr=$POD_CIDR --node-name $NODENAME
   

2. once done we can see the output as follows:

    Your Kubernetes control-plane has initialized successfully!
   
    To start using your cluster, you need to run the following as a regular user:
   
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
   
    Alternatively, if you are the root user, you can run:
   
      export KUBECONFIG=/etc/kubernetes/admin.conf
   
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
   
    Then you can join any number of worker nodes by running the following on each as root:
   
    kubeadm join 192.168.1.95:6443 --token rt9hbl.rm42uzvezo1h76ms \
            --discovery-token-ca-cert-hash sha256:11bfcff27dff2930b0f6a236d10105d721239334aa65c738e4a411b7e779ea38
   

   as you can see we have a few things to do after installation:

   • if running as a regular user, run the following:

       mkdir -p $HOME/.kube
       sudo cp -i /etc/kubernetes/adm2in.conf $HOME/.kube/config
       sudo chown $(id -u):$(id -g) $HOME/.kube/config
     

     this allows us to run kubectl commands from the host with this config file. We can copy this config into our personal PC if we do not want to SSH into the control plane to apply changes.

   • on the other virtual machine that we have cloned, we can join the master node as follows:

       # update worker hostname to k8-worker-1
       sudo hostnamectl set-hostname k8-worker-1
       sudo reboot
     
       # join network
       kubeadm join 192.168.1.95:6443 --token rt9hbl.rm42uzvezo1h76ms \
               --discovery-token-ca-cert-hash sha256:11bfcff27dff2930b0f6a236d10105d721239334aa65c738e4a411b7e779ea38
     

     Note that you should use your token as shown in your terminal instead of the token shown here. this is an example of what it may look like.

   • This is what it should look like after joining successfully

       This node has joined the cluster:
       * Certificate signing request was sent to apiserver and a response was received.
       * The Kubelet was informed of the new secure connection details.
     
       Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
     

3. Here are some kubectl commands to verify the cluster

    kubectl get pod -n kube-system
    kubectl get nodes
    kubctl cluster-info
    kubectl get --raw='/readyz?verbose'
   

Installing Calico Network Plugin

The kubeadm installation also mentions deploying a pod network to the cluster. We will be using Calico to manage our pods' networking and policy.

documentation on Calico can be found here

# install Tigera Calico operator and CRDs
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml
# install Calico and CRDs
curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/custom-resources.yaml > calico.yaml
# update ipPools.cidr: 192.168.0.0/16 to your defined $POD_CIDR above
kubectl create -f calico.yaml

# ensure the pods are running
watch kubectl get pods -n calico-system
kubectl get nodes -o wide

The output of kubectl get pods -A should be as shown below

Every 2.0s: kubectl get pods -A                                                      k8-master: Sun Feb 11 07:06:16 2024

NAMESPACE          NAME                                      READY   STATUS    RESTARTS   AGE
calico-apiserver   calico-apiserver-567f687f88-dkhkr         1/1     Running   0          2m8s
calico-apiserver   calico-apiserver-567f687f88-sftwl         1/1     Running   0          2m8s
calico-system      calico-kube-controllers-6c5c88c78-j9n57   1/1     Running   0          3m39s
calico-system      calico-node-49pjm                         1/1     Running   0          3m39s
calico-system      calico-node-xdkgv                         1/1     Running   0          3m39s
calico-system      calico-typha-6656cb9b9b-dsgnc             1/1     Running   0          3m39s
calico-system      csi-node-driver-4xlkj                     2/2     Running   0          3m39s
calico-system      csi-node-driver-c5s9d                     2/2     Running   0          3m39s
kube-system        coredns-5dd5756b68-bg8jz                  1/1     Running   0          56m
kube-system        coredns-5dd5756b68-ghrtb                  1/1     Running   0          56m
kube-system        etcd-k8-master                            1/1     Running   0          56m
kube-system        kube-apiserver-k8-master                  1/1     Running   0          57m
kube-system        kube-controller-manager-k8-master         1/1     Running   0          56m
kube-system        kube-proxy-jhdds                          1/1     Running   0          56m
kube-system        kube-proxy-sjmrl                          1/1     Running   0          36m
kube-system        kube-scheduler-k8-master                  1/1     Running   0          56m
tigera-operator    tigera-operator-55585899bf-ktqsx          1/1     Running   0          3m50s

Simple Deployment

Now that our Kubernetes cluster is set up, let's run a simple Nginx deployment to verify our cluster is working

First, create a nginx-deployment.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 31000

This will create one Nginx deployment and also its corresponding service, which we will expose externally via nodePort on port 31000

Run kubectl apply -f nginx-deployment.yaml to apply our changes. If everything goes well, we should be able to see our default Nginx page at http://<node IP>:31000

Resources referenced:

• https://kubernetes.io/

• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

• https://github.com/cri-o/cri-o/blob/main/install.md#install-packaged-versions-of-cri-o

• https://docs.tigera.io/calico/latest/getting-started/kubernetes/quickstart

Now we have a bunch of docker containers set up in our internal Linux container. They can be accessed through the IP address of the Linux container, 192.168.1.4 and the respective port number. However, this seems a bit troublesome whenever we need to type it in our web browser URL. Instead, what we will do is create a nginx docker container to handle and redirect all our oncoming network coming in. This nginx will act as a reverse proxy to our backend applications.

what is a reverse proxy?

A reverse proxy is a device or application that can handle requests from clients and redirect the request to other backend services that are managed by us. Our reverse proxy acts as the middleman between the clients and the backend servers. From the point of the client, we are only accessing from a single IP address which is the reverse proxy. The client will not be able to directly access our backend servers' IP address and ports, which improves our security.

Some other notable functions of a reverse proxy include:

1. load balancing

   The reverse proxy can distribute the request between the list of upstream servers provided. This ensures that servers will not be overloaded with requests, which will help our system's stability and latency.

   Some reverse proxies can also provide health checks that periodically check the uptime of the upstream server. If the reverse proxy discovers that one of the servers is down, then it can help to redirect the requests to other available servers.

2. caching

   Request can be cached in the reverse proxy for static content to reduce unnecessary queries to the backend for the same content. This helps to reduce the load on our backend servers.

3. SSL encryption

   SSL can be decrypted at the reverse proxy to read the unencrypted data. This data can then be passed to our backend servers without the use of SSL. That way our backend servers will not need to do the decryption themselves, which can free up valuable resources

let's set up our own reverse proxy

Check out the applications we deployed previously here.

The reverse proxy that we will be using is Nginx. Instead of accessing each of our applications deployed previously via their IP address and port number, we will use easy to remember domain names to access our application

1. Firstly let us configure our DNS server to add the entry in our local DNS record to resolve the domain name to the IP address.

   we will resolve dashy.local, gitea.local, paperless.local, seafile.local to our host machine IP address at 192.168.1.4.

   I am doing this in my own Pi-hole server under Local DNS records. This configuration can also be done on the router which manages the DNS records.

   The Pi-hole example is shown below:

   

   Be sure to add both www.XXX.local and XXX.local domains in the records.

2. Now we can set up a Nginx reverse proxy in our docker environment that listens to all incoming connections to the host machine.

   Write a docker-compose.yml file with the contents as shown below:

    version: "1.0"
    services:
      nginx:
        image: nginx:stable
        container_name: nginx
        volumes:
          - /home/internal/nginx/nginx.conf:/etc/nginx/conf.d/nginx.conf
          - /home/internal/nginx/access.log:/var/log/nginx/access.log
          - /home/internal/nginx/error.log:/var/log/nginx/error.log
        ports:
          - "80:80"
   

   Here are some explanations of this ymal file:

   image:

   We defined a nginx service that uses the nginx:stable image to run our application.

   container_name

   container_name here helps to give our container an easy name to refer to.

   volumes:

   volumes here we have nginx.conf with our reverse proxy configuration, we will be writing later. We also have the access.log and error.log linked from our local storage to the container's default log location for nginx to refer to log entries more easily.

   ports:

   Ports here help to expose the ports in our container to our LXC container. 80:80 , the right-hand side refers to the port in the container and nginx, while the left-hand side refers to the port on the host machine.

3. Writing our nginx configurations

    server {
        listen 80;
        server_name www.gitea.local gitea.local;
   
        location / {
            proxy_pass http://192.168.1.4:3000;
   
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
   
    server {
        listen 80;
        server_name www.paperless.local paperless.local;
   
        location / {
            proxy_pass http://192.168.1.4:8000;
   
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
   
    server {
        listen 80;
        server_name www.dashy.local dashy.local;
   
        location / {
            proxy_pass http://192.168.1.4:4000;
   
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
   
    server {
        listen 80;
        server_name www.seafile.local seafile.local;
   
        location / {
            proxy_pass http://192.168.1.4:3001;
   
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
   

   Some explanation:

   listen keyword here denotes that nginx is listening to port 80 in the container.

   server_name here refers to the domain that nginx is handling, which is important for our domain-based routing.

   location / block here denotes how the root URL is being handled.

   proxy_pass here means nginx will forward the incoming request to the specified URL instead, which in this case is our application IP address and port number.

   proxy_set_header here change the headers of our incoming request so that the application knows where did this request originally came from. For example, X-Forwarded-For header here is the chain of IP addresses starting from the client IP address all the way to the last proxy IP address before reaching the application. There can be several levels of proxies before it eventually reaches our application.

4. start our nginx server by running docker compose up -d

5. try navigating to our applications via dashy.local, seafile.local, gitea.local or paperless.local and verify everything is running all right.

We have now implemented our very own reverse proxy 🎉

Here is what it looks like before:

Here is our new updated network flow:

what's next?

In the future, we will be adding SSL security to our nginx so that we can connect via HTTPS instead of HTTP.

sudo apt update && apt upgrade
apt install curl

let's install docker now using the quick install script

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh ./get-docker.sh

now let's create our non-privileged user and call it internal

we wouldn't want to run docker with a root user for security issues. If a user in the docker container managed to break out of it, it may gain root access privileges on the host.

we also need to add this user to the docker group to have access to the docker commands

sudo adduser internal
sudo usermod -aG docker internal

now we can set up a few applications that we can use

For easier access to application files, we will be using docker bind mounts instead of named volume mounts. We will also want the data folders to be located in each of the respective application folders.

• set up gitea

  Gitea is an application that allows us to host our git repositories

  create a gitea directory and add a docker-compose.yml file. we will be using postgresql for this app.

    version: "3"
  
    networks:
      gitea:
        external: false
  
    services:
      server:
        image: gitea/gitea:1.21.0
        container_name: gitea
        environment:
          - USER_UID=1000
          - USER_GID=1000
          - GITEA__database__DB_TYPE=postgres
          - GITEA__database__HOST=db:5432
          - GITEA__database__NAME=gitea
          - GITEA__database__USER=gitea
          - GITEA__database__PASSWD=gitea
        restart: always
        networks:
          - gitea
        volumes:
          - /home/internal/gitea/data:/data
          - /etc/timezone:/etc/timezone:ro
          - /etc/localtime:/etc/localtime:ro
        ports:
          - "3000:3000"
          - "222:22"
        depends_on:
          - db
  
      db:
        image: postgres:14
        restart: always
        environment:
          - POSTGRES_USER=gitea
          - POSTGRES_PASSWORD=gitea
          - POSTGRES_DB=gitea
        networks:
          - gitea
        volumes:
          - /home/internal/gitea/postgres:/var/lib/postgresql/data
  

  start the containers with docker compose up -d and verify if the link is working at http://<ip>:3000

• set up paperless-ngx

  Paperless-ngx is a document management system that can archive and index important documents so we do not have extra paper or receipts lying around

  we can run an easy install script that will auto create our folders and configurations

  since we are using volume bind mounts, be sure to set the consume, media and db folders to use absolute paths

    bash -c "$(curl -L https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/main/install-paperless-ngx.sh)"
  

  verify if the link is working at http://<ip>:8000

• set up dashy

  Dashy is a homepage organizer to keep tracks of all the links that we will have when we create more applications down the road

  create a dashy directory and add a docker-compose.yml file.

    ---
    version: "3.8"
    services:
        dashy:
          image: lissy93/dashy
          container_name: Dashy
          # Pass in your config file below, by specifying the path on your host machine
          volumes:
            - /home/internal/dashy/data/my-config.yml:/app/public/conf.yml
          ports:
            - 4000:80
          # Set any environmental variables
          environment:
            - NODE_ENV=production
          # Specify your user ID and group ID. You can find this by running `id -u` and `id -g`
            - UID=1000
            - GID=1000
          # Specify restart policy
          restart: unless-stopped
          # Configure healthchecks
          healthcheck:
            test: ['CMD', 'node', '/app/services/healthcheck']
            interval: 1m30s
            timeout: 10s
            retries: 3
            start_period: 40s
  

  start the containers with docker compose up -d and verify if the link is working at http://<ip>:4000

• set up seafile

  Seafile is a file hosting system that can keep track of important files and data. Files can be sync across multiple devices as well.

  create a seafile directory and add a docker-compose.yml file.

    services:
      db:
        image: mariadb:10.11
        container_name: seafile-mysql
        environment:
          - MYSQL_ROOT_PASSWORD=db_dev  # Requested, set the root's password of MySQL service.
          - MYSQL_LOG_CONSOLE=true
        volumes:
          - /home/internal/seafile/mysql/db:/var/lib/mysql  # Requested, specifies the path to MySQL data persistent store.
        networks:
          - seafile-net
  
      memcached:
        image: memcached:1.6.18
        container_name: seafile-memcached
        entrypoint: memcached -m 256
        networks:
          - seafile-net
  
      seafile:
        image: seafileltd/seafile-mc:11.0.0
        container_name: seafile
        ports:
          - "80:80"
    #     - "443:443"  # If https is enabled, cancel the comment.
        volumes:
          - /home/internal/seafile/data:/shared   # Requested, specifies the path to Seafile data persistent store.
        environment:
          - DB_HOST=db
          - DB_ROOT_PASSWD=db_dev  # Requested, the value should be root's password of MySQL service.
          - TIME_ZONE=Asia/Singapore  # Optional, default is UTC. Should be uncomment and set to your local time zone.
          - SEAFILE_ADMIN_EMAIL=me@example.com # Specifies Seafile admin user, default is 'me@example.com'.
          - SEAFILE_ADMIN_PASSWORD=PASSWORD     # Specifies Seafile admin password, default is 'asecret'.
          - SEAFILE_SERVER_LETSENCRYPT=false   # Whether to use https or not.
    #      - SEAFILE_SERVER_HOSTNAME=docs.seafile.com # Specifies your host name if https is enabled.
        depends_on:
          - db
          - memcached
        networks:
          - seafile-net
  
    networks:
      seafile-net:
  

  start the containers with docker compose up -d and verify if the link is working at http://<ip>:80

Let's look at our homelab overview again

I used a static IP address for my internal Linux container and set it to 192.168.1.4 for easy reference when I want to enter into my web app. This can be changed inside /etc/network/interfaces for Ubuntu.

Manually typing http://192.168.1.4:4000 can be difficult to remember. In the future, we can add a reverse proxy to handle all the connections to our internal docker containers.

links to the application documentation

• https://docs.gitea.com/

• https://docs.paperless-ngx.com/setup/

• https://dashy.to/docs/quick-start/

• https://manual.seafile.com/docker/deploy_seafile_with_docker/

Recently I got my hands on an old server desktop. I plan to repurpose it into a home lab to host some internal applications to be used at home and also host some of my side projects which can be externally accessed by the public. I have decided to try installing Proxmox on the desktop.

Proxmox is an open-source virtualization management platform that can provide both Kernel-based Virtual machines (KVM) and also container-based virtualization with Linux Containers (LXC). It also provides a nifty web-based interface for the user to interact with Proxmox.

Hardware specs

Here is some hardware information about my old server desktop:

• Precision Tower 5810

• Intel Xeon E5-1607 v3

• Nvidia Quadro K2200

• 8GB of ram

• 256GB Samsung SSD

• 500GB Hard disk drive

It's a pretty old desktop, released in 2014 and I managed to get all its components for free. Next, we can try to wipe everything in the disk drives and install Proxmox.

Installing Proxmox

1. The Proxmox VE (Virtual Environment) can be downloaded for free from the official Proxmox website. The version I am using is proxmox-ve_7.3-1.iso.

2. Create a bootable installation media from a USB drive. Here I used Rufus to write the Proxmox ISO image.

   

   For Rufus, remember to write in DD Image or else it would not work.

   

3. Insert the USB drive into the desktop and boot from the USB drive.

   • remember to disable secure boot on the BIOS of certain Desktop

   • you might need to enter the BIOS to change the boot order so that the USB drive can be booted first

4. Follow the installation guide shown on the desktop

   1. Select install proxmox ve

   2. Accept End User License Agreement

   3. Select which disk to be used for installing Proxmox

   4. Select password, language and keyboard layout

   5. Configure the management network settings. Here my router is on 192.168.1.254, which I will be using as my gateway and DNS server. I set the IP address of my Proxmox server to 192.168.1.2 for easy reference. In the router web interface, I also assigned a static IP address of 192.168.1.2 to the Mac address of my desktop.

      • Hostname: host1.pve.local

      • IP address: 192.168.1.2

      • Gateway: 192.168.1.254

      • DNS server: 192.168.1.254

   6. After installation is successful, you will be prompted to reboot.

Accessing Proxmox Web Interface

Once fully installed, we can go to the Proxmox web interface denoted by the IP address of the Proxmox server and run on port 8006. In my case, it would be at https://192.168.1.2:8006. The interface is shown below.

Currently, it only has one node, which represents a physical server that is currently running. We could install another Promox on another desktop to have another node, but one should be enough for a small home lab.

Creating Linux Containers

Now, we are ready to create our Linux Containers.

I want to use Linux Containers (LXC) over Kernel-based Virtual machines (KVM) because it provides lightweight virtualization and efficient resource utilization. KVM runs its own kernel as compared with LXC which uses the host operating system's kernel and shares them with other containers. This means that there is less overhead and fewer resources utilized.

The downside of LXC is that since our host operating system uses Linux, LXC can only run Linux-based operating systems such as Ubuntu or ArchLinux.

Let's get some LXC templates first. Under host1 > local > CT Templates > Templates, choose ubuntu-22.04-standard and then press download.

then click on Create CT on the top right-hand side of the interface. Set our Node as host1, our physical server and the hostname of our LXC as internal. I intend for this LXC to contain all my docker applications that are used internally in my home and not exposed to the public.

Next, select the template we had just downloaded earlier.

Set the amount of disk space we require.

Followed by the size of the ram we require.

Lastly, we configure the IPv4 and IPv6 to use DHCP, to dynamically allocate IP addresses for our new LXC.

Confirm our selection and then reboot. Once our LXC has started up, we can view it through the console tab as shown below.

I will set up 2 more containers, one that is externally accessible for public use, named external, and the other for development use, named dev.

Architecture

Here is an overview of what I have created.

Conclusion

I have installed Proxmox on an old desktop and launched three Linux containers. In the future, I plan to install applications that I can use in my own home network and also to host some of my projects to the public.

In the app that we are creating today, we will be using NextJs for our frontend, Go for our backend and saving our user data into MongoDB.

The full code can be found on my GitHub repository.

Prerequisites

Before diving into the implementation, make sure to have the following prerequisites:

1. Go installed on your machine

2. Node.js and NPM installed on your machine

3. MongoDB operations in Go (can read the documentation here)

4. A MongoDB database set up and accessible (I am using MongoDB Atlas which is exposed using a MONGODB_URI)

5. An email service provider (here I am using Gmail to send my emails)

Architecture

Here is a general overview of the components we will be building and the connection between the different services. We will only be sending our crafted emails to our mail service server using SMTP protocol, which will then forward this email to the intended recipient.

Approach

After a user registers for an account, we want to store their details in our database and then send them an email with a verification link and code.

For our approach to verification with a link, we want to generate a link such that when clicked, will make a GET request to our server and once verified, we will redirect them back to our frontend homepage for them to log in.

For our approach to verification with code, we want to generate a random 4-digit code that is sent to their email. The user can then key this code in the frontend after they registered and if the code is correct, will allow the user into the dashboard.

Configure SMTP with mail service

The email service provider I am using is Gmail and they have a guide on how to enable SMTP on the Gmail account. You can use other options like Sendgrid, Mailgun or Outlook.

For Gmail, we need to enable the IMAP Access option on our Gmail settings page.

We will also need to generate an App Password as we do not want to use the same password we use for our Google account to authenticate our SMTP access. For that, we can go to Google account settings, under Security and then 2 steps verification, add your new app password.

Setting up backend service

1. Make backend folder

    mkdir backend
    cd backend
   

2. Set up a new Go module and install necessary dependencies

    go mod init auth-boilerplate
   
    go get github.com/gin-gonic/gin
    go get github.com/gin-contrib/cors
   
    go get go.mongodb.org/mongo-driver/mongo
    go get github.com/joho/godotenv
   
    go get golang.org/x/crypto/bcrypt
   

3. Set up the API server using gin and include cors settings so that we do not run into any "Blocked by CORS policy" errors.

    func main() {
        r := gin.Default()
   
        r.Use(cors.New(cors.Config{
            AllowOrigins:  []string{fmt.Sprintf("http://%s", origin)},
            AllowMethods:  []string{"POST", "GET", "PUT", "OPTIONS", "DELETE"},
            AllowHeaders:  []string{"Origin", "Content-Type", "Set-Cookie"},
            ExposeHeaders: []string{"Content-Length", "Content-Type", "Set-Cookie", "Access-Control-Allow-Credentials", "Access-Control-Expose-Headers", "Access-Control-Allow-Origin", "set-cookie"},
   
            AllowCredentials: true,
            AllowWebSockets:  true,
            MaxAge:           12 * time.Hour,
        }))
   
        r.POST("/verify", verifyCode)
        r.GET("/verify/link", verifyLink)
        r.POST("/register", register)
        r.POST("/login", login)
   
        r.Run(":3001")
    }
   

4. Establish a connection with the Mongo database using mongo.Connect() and if successful, we can use this Mongo client to query and insert our collections. We will use godotenv.Load(".env") to load our MONGODB_URI value from a .env file in our project folder.

    var client *mongo.Client
   
    func initDatabase() error {
        uri := os.Getenv("MONGODB_URI")
        fmt.Println(uri)
        if uri == "" {
            log.Println("URI not found in environmental variables")
            return errors.New("URI not found")
        }
   
        var err error
        client, err = mongo.Connect(context.TODO(), options.Client().ApplyURI(uri))
        if err != nil {
            log.Println("Error connecting to database")
            return errors.New("cannot connect to db")
        }
   
        return nil
    }
   
    func main() {
        err := godotenv.Load(".env")
        if err != nil {
            log.Println("failed to load .env file")
            return
        }
        err = initDatabase()
        if err != nil {
            return
        }
        ///...
    }
   

5. Declare user struct to handle user inputs from JSON and BSON. BSON, which stands for binary JSON, is used by MongoDB to store and transmit data. When omitempty is used in the ID field, it means that the ID field will not be added into MongoDB if the value of ID in our User struct is empty, MongoDB will auto-generate the _id field for us.

   We will be generating the VerifyHash and VerifyCode on Registration and compare them when they try to verify.

   ValidHash and ValidCode will be storing the deadline when the hash or code is considered invalid and will need to be regenerated.

    type User struct {
        ID          string    `bson:"_id,omitempty"`
        Name        string    `json:"name" bson:"name"`
        Email       string    `json:"email" bson:"email"`
        Password    string    `json:"password" bson:"passwordHash"`
        VerifyHash  string    `bson:"verifyhash"`
        VerifyCode  string    `bson:"verifycode"`
        IsVerified  bool      `json:"isVerified" bson:"isVerified"`
        DateCreated time.Time `bson:"dateCreated"`
        ValidHash   time.Time `bson:"validHash"`
        ValidCode   time.Time `bson:"validCode"`
    }
   

6. Let's add our register function whenever the user registers on the frontend.

   ctx.BindJSON is used to bind the JSON data from the request body to the newUser struct.

   bcrypt.GenerateFromPassword function is used to generate a secure hash from the user's password. We then replace the current password provided by the user.

   We will then generate the verification link and code and add them to the struct, together with their expiration.

   We will insert our newUser struct into the database using the InsertOne method and add it to our auth database and users collection.

   Lastly, we will send the user their verification email with the link and code attached.

    func register(ctx *gin.Context) {
        var newUser User
   
        err := ctx.BindJSON(&newUser)
        if err != nil {
            log.Println("Error binding JSON")
            ctx.AbortWithStatusJSON(http.StatusInternalServerError, nil)
            return
        }
   
        // ensure email does not exist in database
        // ensure email and password is valid
        // left as an exercise
   
        // generate hash from user password
        hash, err := bcrypt.GenerateFromPassword([]byte(newUser.Password), 6)
        if err != nil {
            log.Println("error hashing password")
        }
        newUser.Password = string(hash)
   
        // generate verification link
        newUser.VerifyHash = generateVerifyHash()
        // generate verification code
        newUser.VerifyCode = generateVerifyCode()
   
        // add deadline for hash and code expiry
        // in the example here, we use 15mins for hash and 3 mins for code
        now := time.Now()
        newUser.DateCreated = now
        newUser.ValidHash = now.Add(15 * time.Minute)
        newUser.ValidCode = now.Add(3 * time.Minute)
   
        // add user to database
        coll := client.Database("auth").Collection("users")
        log.Println("add: ", newUser)
        result, err := coll.InsertOne(context.TODO(), newUser)
        if err != nil {
            log.Println("failed to add new user:", err)
            ctx.JSON(http.StatusOK, gin.H{
                "success": false,
            })
            return
        }
   
        fmt.Println(result)
   
        // send the verification email
        sendVerificationEmail(newUser)
   
        ctx.JSON(http.StatusOK, gin.H{
            "success": true,
        })
    }
   

7. Create our login function.

   We will use the user's email to query the database and fetch their document.

   Once found, use bcrypt.CompareHashAndPassword to compare the hash in the database and the user's password in the request body. If successful, then proceed to return a JSON response based on whether the user is verified. If not successful, then return a JSON response with an HTTP code unauthorized.

    func login(ctx *gin.Context) {
        var user User
   
        err := ctx.BindJSON(&user)
        if err != nil {
            log.Println("Error binding JSON")
            ctx.AbortWithStatusJSON(http.StatusInternalServerError, nil)
            return
        }
   
        var dbUser User
        coll := client.Database("auth").Collection("users")
        filter := bson.D{{"email", user.Email}}
        err = coll.FindOne(context.TODO(), filter).Decode(&dbUser)
        if err != nil {
            if err == mongo.ErrNoDocuments {
                log.Println(fmt.Sprintf("searching for users with email: %s yield no result found", dbUser.Email))
                ctx.JSON(http.StatusUnauthorized, gin.H{
                    "success": false,
                    "error":   "Either email or password is incorrect",
                })
                return
            }
            log.Println("failed to search for user")
        }
   
        err = bcrypt.CompareHashAndPassword([]byte(dbUser.Password), []byte(user.Password))
        if err != nil {
            log.Println("password is different")
            ctx.JSON(http.StatusUnauthorized, gin.H{
                "success": false,
                "error":   "Either email or password is incorrect",
            })
            return
        }
   
        if !user.IsVerified {
            ctx.JSON(http.StatusOK, gin.H{
                "success":    false,
                "isVerified": false,
                "error":      "user not verified",
            })
        }
   
        ctx.JSON(http.StatusOK, gin.H{
            "success":    true,
            "isVerified": false,
            "error":      "",
        })
    }
   

8. Let's create our handler function for verifyCode and verifyLink.

   We will read the link's email, hash and code from the request's query params. Compare the hash or code with the one retrieved from the database. If successful, we can then update the isVerified field in the Mongo database.

   Here we use the coll.UpdateOne to update our document based on the _id of the document.

    func verifyCode(ctx *gin.Context) {
        // read from query params
        email := ctx.Query("email")
        hash := ctx.Query("hash")
        code := ctx.Query("code")
   
        var user User
   
        coll := client.Database("auth").Collection("users")
        filter := bson.D{{"email", email}}
        err := coll.FindOne(context.TODO(), filter).Decode(&user)
        if err != nil {
            if err == mongo.ErrNoDocuments {
                log.Println(fmt.Sprintf("searching for users with email: %s yield no result found", email))
            }
            log.Println("failed to search for user")
        }
   
        // set user to verified
        if user.VerifyCode == code && user.ValidCode.After(time.Now()) {
            id, _ := primitive.ObjectIDFromHex(user.ID)
            filter = bson.D{{"_id", id}}
            update := bson.D{{"$set", bson.D{{"isVerified", true}}}}
            result, err := coll.UpdateOne(context.TODO(), filter, update)
            if err != nil {
                log.Println("failed to update:", err)
            }
            log.Println(result)
            ctx.JSON(http.StatusOK, gin.H{
                "success": true,
            })
            return
        }
   
        ctx.JSON(http.StatusOK, gin.H{
            "success": false,
            "error":   "incorrect code",
        })
    }
   

9. Let's implement the helper functions we used earlier.

   generateVerifyCode will randomly generate an Int between 0 to 9999 and then pad the value such that it is 4 characters long

   generateVerifyHash will randomly generate random bytes in our array and then change this value by encoding it into a string via base64 encoding.

    func generateVerifyCode() string {
        rand.Seed(time.Now().UnixNano())
        randomNumber := rand.Intn(10000)
        paddedNumber := fmt.Sprintf("%04d", randomNumber)
   
        return paddedNumber
    }
   
    func generateVerifyHash() string {
        charLen := 50
        randomBytes := make([]byte, charLen)
   
        _, err := rand.Read(randomBytes)
        if err != nil {
            fmt.Println("failed to generate random bytes")
        }
   
        randomString := base64.URLEncoding.EncodeToString(randomBytes)
        return randomString[:charLen]
    }
   

10. Set up our connection with the SMTP host. We then use the net/smtp package from Go to construct and send our email.

    const (
        // configs to connect to gmail smtp server
        smtpHost = "smtp.gmail.com"
        smtpPort = 587
    )
    
    appkey = os.Getenv("APPKEY")
    hostEmail = os.Getenv("HOSTEMAIL")
    
    func getVerifyLink(email string, verifyHash string, verifyCode string) string {
        return fmt.Sprintf("http://localhost:3001/verify/link?email=%s&hash=%s&code=%s", email, verifyHash, verifyCode)
    }
    
    func sendVerificationEmail(user User) {
        verifyLink := getVerifyLink(user.Email, user.VerifyHash, user.VerifyCode)
        to := []string{user.Email}
        subject := "Email verification"
        body := fmt.Sprintf("Hello, \r\n\r\nThis email contains the verification link and verification code, please the link below to verify your account.\r\n\r\nVerification link: %s\r\n\r\nAlternatively, u can also key in the verification code to verify.\r\n\r\nVerification code: %s", verifyLink, user.VerifyCode)
    
        message := []byte("To: " + user.Email + "\r\n" +
            "From: " + hostEmail + "\r\n" +
            "Subject: " + subject + "\r\n" +
            "\r\n" +
            body + "\r\n")
    
        auth := smtp.PlainAuth("", hostEmail, appkey, smtpHost)
    
        err := smtp.SendMail("smtp.gmail.com:587", auth, hostEmail, to, message)
        if err != nil {
            log.Println("Failed to send email")
            log.Println(err)
            return
        }
        log.Println("Email sent successfully")
    }
    

Setting up frontend

1. Initialize a new Next.js project using the command below and name the folder frontend

    npx create-next-app@latest
   

2. In the root frontend, create a folder components that holds our styled-components for Fields, Buttons and ButtonText.

    // component.tsx
   
    export const Field = ({ title, placeholder = '', type = 'text', value, onChange }:
        { title: string, placeholder?: string, type?: string, value: string, onChange: Function }) => {
        return (
            <div>
                <h3>{title}</h3>
                <input className="py-2 px-4 leading-tight text-gray-700 appearance-none border-4 border-gray-200 rounded focus:outline-none focus:bg-white focus:border-teal-500"
                    placeholder={placeholder} type={type} value={value} onChange={(e) => onChange(e.target.value)} />
            </div>
        )
    }
   
    export const Button = ({ text, onClick }: { text: string, onClick?: Function }) => {
        return (
           <button onClick={() => { onClick?.() }} 
                className="px-7 py-3 mt-10 bg-teal-500 hover:bg-teal-700 border rounded-lg font-bold">{text}</button>
        )
    }
   
    export const ButtonText = ({ text, onClick }: { text: string, onClick: Function }) => {
        return (
            <button onClick={() => { onClick() }} 
                className="px-7 py-3 mt-10 hover:bg-gray-600 rounded-lg font-bold">{text}</button>
        )
    }
   

3. Under /app/page.tsx, replace the boiler code with our login, register and Verify components. They provide the fields for inputting our user's login or register info and calling the backend API on submit.

    'use client'
   
    import { useState } from "react"
    import { useRouter } from "next/navigation"
    import { Button, ButtonText, Field } from "@/components/component"
   
    export default function Home() {
        const [isLogin, setLogin] = useState(true)
        const [showVerify, setVerify] = useState(false)
        const [email, setEmail] = useState("")
   
        return (
            <main className="min-h-screen flex flex-col items-center justify-center">
                <h1 className="text-4xl font-bold mb-10">Authentication demo page</h1>
                {
                    showVerify ? <Verify email={email}/> : isLogin 
                        ? <Login setLogin={setLogin} setVerify={setVerify}/>
                        : <Register setLogin={setLogin} setVerify={setVerify} email={email} setEmail={setEmail}/>
                }
            </main>
        )
    }
   
    function Register({ setLogin, setVerify, email, setEmail }:
        { setLogin: Function, setVerify: Function, email: string, setEmail: Function }) {
        const [name, setName] = useState("")
        const [password, setPassword] = useState("")
        const [repassword, setRepassword] = useState("")
        const [error, setError] = useState("")
   
        const register = () => {
            if (password !== repassword) {
                return
            }
            const user = {
                name,
                email,
                password
            }
            fetch("http://localhost:3001/register", {
                method: "POST",
                headers: {
                    "Content-Type": "application/json",
                },
                body: JSON.stringify(user)
            })
                .then((resp) => resp.json())
                .then(({ success, error }) => {
                    if (success) {
                        setVerify(true);
                    } else {
                        // handle incorrect login 
                        setError(error)
                    }
                })
        }
   
        return (
            <>
                <h1 className="text-4xl font-bold mb-10">Register</h1>
                <div className="flex flex-col gap-5">
                    <Field title={'Name'} value={name} onChange={setName} />
                    <Field value={email} onChange={setEmail}
                        title={'Email'} placeholder="example@mail.com" type="email" />
                    <Field value={password} onChange={setPassword}
                        title={'Password'} type="password" />
                    <Field value={repassword} onChange={setRepassword}
                        title={'Retype Password'} type="password" />
                </div>
                <Button text={"Register"} onClick={register} />
                <ButtonText text={"Login instead"} onClick={() => setLogin(true)} />
            </>
        )
    }
   
    function Login({ setLogin, setVerify }: { setLogin: Function , setVerify: Function }) {
        const router = useRouter()
   
        const [email, setEmail] = useState("")
        const [password, setPassword] = useState("")
        const [error, setError] = useState("")
   
        const login = () => {
            const user = {
                email,
                password
            }
   
            fetch("http://localhost:3001/login", {
                method: "POST",
                headers: {
                    "Content-Type": "application/json",
                },
                body: JSON.stringify(user)
            })
                .then((resp) => resp.json())
                .then(({ success, isVerified, error }) => {
                    if (success) {
                        if (isVerified) {
                            router.push("/dashboard")
                        }
                        setVerify(true);
                    } else {
                        // handle incorrect login 
                        setError(error)
                    }
                })
        }
   
        return (
            <>
                <h1 className="text-4xl font-bold mb-10">Login</h1>
                <div className="flex flex-col gap-5">
                    <Field value={email} onChange={setEmail}
                        title={'Email'} placeholder="example@mail.com" type="email" />
                    <Field value={password} onChange={setPassword}
                        title={'Password'} type="password" />
                </div>
                {error === "" ? <></> : <p className="my-2 text-red-800">{error}</p>}
                <Button text={"Login"} onClick={() => { login() }} />
                <ButtonText text={"Register instead"} onClick={() => setLogin(false)} />
            </>
        )
    }
   
    function Verify({ email }: {email: string}) {
   
        const router = useRouter()
        const [code, setCode] = useState("")
        const [error, setError] = useState("")
   
        const verify = () => {
            fetch(`http://localhost:3001/verify?email=${email}&code=${code}`, {
                method: "POST",
                headers: {
                    "Content-Type": "application/json",
                },
            })
                .then((resp) => resp.json())
                .then(({ success, error }) => {
                    if (success) {
                        router.push("/dashboard")
                    } else {
                        // handle incorrect login 
                        setError(error)
                    }
                })
        }
   
        return (
            <div className="flex flex-col gap-5 items-center justify-center">
                <h1 className="uppercase text-xl" >Verify your email address</h1>
                <div className="w-1/2 text-center">
                <p className="">A verification code and link has been sent to *****@****mail.com</p>
                </div>
                <p>Please check your inbox or junk folder and proceed to verify with us. The code will expire in 2mins</p>
                <Field title="Code" value={code} onChange={setCode} />
                <Button text={"Verify"} onClick={() => {verify()}} />
            </div>
        )
    }
   

4. Lastly, we create the dashboard route when the user successfully logins and also the verify route when the user clicks on the verify link which will prompt them to return to the homepage to log in again.

    // in /app/dashboard/page.tsx
    "use client"
    export default function Page() {
        return (
            <div className="min-h-screen flex flex-col items-center justify-center">
                <h1 className="text-3xl mb-3">This is a dashboard viewable when logged in</h1>
                <h2>Return to root page using button below</h2>
                <Link href="/">
                    <Button text="Return"/>
                </Link>
            </div>
        )
    }
   

    // in /app/verify/page.tsx
    "use client"
    export default function Page() {
        return (
            <div className="min-h-screen flex flex-col items-center justify-center">
                <h1 className="text-3xl mb-3">You are now verified!</h1>
                <h2>Click below to return back to the homepage and login</h2>
                <Link href="/">
                    <Button text="Return"/>
                </Link>
            </div>
        )
    }
   

Testing our app

To start our Go server, run

go run .

ensure that you have the .env file in the root of the backend folder. Be sure not to commit this into the git repository as the keys are private and should be kept hidden.

// .env file
MONGODB_URI="mongodb+srv://<USER>:<PASSWORD>@cluster23.honxmor.mongodb.net/?retryWrites=true&w=majority"
HOSTEMAIL="XXXXX@gmail.com"
APPKEY="ABCDEFGH"

To start our Next.js frontend, run:

yarn dev

Our backend will be running in localhost:3001 while our frontend will be running in localhost:3000.

If we try to register:

We will have a user entry created in MongoDB Atlas

and also received my verification email

which we can proceed to click the link or fill up the verification code on our page

Summary

We have created an application that can send verification emails for users to verify.

In the future, I will try to optimize our backend code so that the frontend side can receive the response from the server faster.

SQLite3 should come installed with most Linux distributions but if it is not installed, you can run sudo apt install sqlite3

Below is the updated file structure we are going to modify today:

API endpoints

The first step is to create a room for the home. Rooms can be any part of the room in the house, for example, bedroom, living room or kitchen. Once our rooms are created, we can start associating the rooms with the devices we want to connect.

Our devices can then be tied to the rooms we created and be listed on our front end.

The table below shows the different API routes we will be creating and their description.

Some of our routes have parameters in their paths as denoted by the : in front of the names. The handler will match the routes based on the parameters and map them into values. So a route such as /bedroom/devices will match with the /:roomname/devices and roomname == 'bedroom'.

We can then translate this into code and add them to our router.go file

// router.go

func InitRouter() *gin.Engine {
    r := gin.Default()

    r.GET("/", getServerStatus)


    r.POST("/create-room", createRoom)
    r.GET("/rooms", getRooms)
    r.DELETE("/delete-room/:roomname", deleteRoom)

    r.POST("/:roomname/add-device", addDevice)
    r.GET("/:roomname/devices", showDevices)

    r.GET("/:roomname/:ip", getDeviceInfo)
    r.DELETE("/:roomname/:ip/delete-device", deleteDevice)
    r.POST("/:roomname/:ip/:toggle", toggleDevice)

    return r
}

creating golang struct

To facilitate the loading of data from our routes as well as our database, we will create golang structs that encapsulate our fields of data together. Under the models folder, we can add the following structs for our room and devices.

// models.go

package models

type RoomInfo struct {
    Name  string `json:"name"`
    Count int    `json:"count"`
}

type DeviceType string

const (
    Wled   DeviceType = "wled"
    Switch DeviceType = "switch"
)

type RegisteredDevice struct {
    Hostname string     `json:"hostname"`
    Ipaddr   string     `json:"ipaddr"`
    Name     string     `json:"name"`
    Type     DeviceType `json:"type"`
}

type DeviceStatus struct {
    Connected bool `json:"connected"`
    On_state  bool `json:"on_state"`
}

All the fields here have a corresponding JSON tag, which specifies how the field should be encoded in JSON.

Type names and Fields are capitalized to indicate that the identifier is public

For RoomInfo, we have a Name field that is unique and the Count field, which we will use to keep track of the number of devices in the room.

To keep track of the device, we have the RegisteredDevice type that has fields about the information of the device such as the name, IP address and the device type. DeviceType is defined as a string type that refers to our constant values wled and switch, which represents the different device types we support.

Our DeviceStatus helps to maintain information about the state of the device. Currently, we have Connected field to check if the device is operational and On_state field to check if the switch is on or off.

setting up our database

We will be using the go-sqlite3 package to help us execute SQL queries to our SQLite3 database.

go get github.com/mattn/go-sqlite3

First, let's create our InitDatabase function that runs every time we start our backend server.

// database.go
package database

import (
    "database/sql"
    _ "github.com/mattn/go-sqlite3"
)

const databaseFilePath = "iothome.db"

func InitDatabase() *sql.DB {
    db, err := sql.Open("sqlite3", databaseFilePath)
    if err != nil {
        log.Println("Error opening database")
    }

    if _, err := db.Exec("PRAGMA foreign_keys = 1"); err != nil {
        log.Println("Error setting command")
    }

    return db
}

Using sql.Open, we can try to open our SQL database using the name that we have given. If no such name exists, then our database driver will initialize a new database file for us.

We then execute the SQL command PRAGMA foreign_keys = 1 to enable foreign key constraints in our database. By default, foreign key constraints are disabled. Foreign key constraints are used to enforce referential integrity, which we will be utilizing in our tables later on.

SQL create tables

Next, we create our database schema for containing our data. We have three different tables, rooms, devInfo and devStatus each with its own headers. In the devInfo table, we have our room_id column that references the room_id column in the rooms table. This means that each device info entry has a room_id column which can be referenced to the rooms table. This is similar to the device_id in devInfo and devStatus.

Let's translate this into SQL commands to create our table, as shown below:

const (
    createRooms string = `
        CREATE TABLE IF NOT EXISTS rooms (
            room_id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
            name TEXT NOT NULL UNIQUE
        )
    `

    createDevInfo string = `
        CREATE TABLE IF NOT EXISTS deviceInfo (
            room_id INTEGER NOT NULL,
            device_id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
            name TEXT NOT NULL,
            ipaddr TEXT NOT NULL,
            type TEXT NOT NULL,
            hostname TEXT,
            macaddress TEXT,
            FOREIGN KEY (room_id)
                REFERENCES rooms (room_id)
                    ON DELETE CASCADE
        )
    `

    createDevStatus string = `
        CREATE TABLE IF NOT EXISTS deviceStatus (
            device_status_id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
            device_id INTEGER NOT NULL,
            connected INTEGER NOT NULL,
            on_state INTEGER NOT NULL,
            FOREIGN KEY (device_id)
                REFERENCES deviceInfo (device_id)
                    ON DELETE CASCADE
        )
    `
)

func InitDatabase() *sql.DB {
    //  ... 
    if _, err := db.Exec(createRooms); err != nil {
        log.Println("failed to create new rooms table")
    }

    if _, err := db.Exec(createDevInfo); err != nil {
        log.Println("failed to create new devinfo table")
    }

    if _, err := db.Exec(createDevStatus); err != nil {
        log.Println("failed to create new devstatus table")
    }
    //  ...
}

We use CREATE TABLE IF NOT EXISTS to initialize our database as it will create the table if the table does not currently exist.

NOT NULL is used here to ensure that the important columns we are storing have a value to them and is not empty.

AUTOINCREMENT keyword is used to create a unique identifier for each of our inserted rows in the table. It automatically generates an integer starting from 1.

ON DELETE CASCASE keywords indicate that when a row in deviceInfo is deleted, all the rows in the deviceStatus table that references the deleted row via the device_id foreign key should also be deleted, which makes sense if we want to delete a particular device from the table.

UNIQUE keyword is used here to ensure that our name column will not have any two rows with the same values.

initializing database

let's create a struct DatabaseManager that holds a pointer to the database that we initialized in the previous section. In our database.go file, we will declare Dbman that holds a pointer to the DatabaseManager. This is so that other files can reference Dbman to execute the SQL commands.

// database.go

type DatabaseManager struct {
    Db *sql.DB
}

var Dbman *DatabaseManager

func InitializeGlobals(db *sql.DB) {
    Dbman = &DatabaseManager{
        Db: db,
    }
}

SQL statements

In database.go, we also declare our SQL commands for inserting, selecting, deleting and updating our entries in the database. The ? here denotes the data that will be replaced when we execute these commands below in our server.

// database.go

const (
    insertNewRoom string = `INSERT INTO rooms (name) VALUES (?)`
    getRooms string = `SELECT rooms.name, count(deviceInfo.room_id) FROM rooms LEFT JOIN deviceInfo ON rooms.room_id = deviceInfo.room_id GROUP BY rooms.name`
    getRoomId     string = `SELECT room_id FROM rooms WHERE name=?`

    insertNewDevInfo string = `INSERT INTO deviceInfo (room_id, name, ipaddr, type) VALUES (?, ? ,? ,?)`
    insertNewDevStatus string = `INSERT INTO deviceStatus (device_id, connected, on_state) VALUES (?, ? ,?)`

    getDeviceInfoByRoom string = `SELECT dev.name, dev.ipaddr, dev.type, dev.connected, dev.on_state FROM rooms JOIN (SELECT * FROM deviceInfo JOIN deviceStatus WHERE deviceInfo.device_id = deviceStatus.device_id) as dev WHERE rooms.room_id = dev.room_id and rooms.name=?`

    getDeviceInfo string = `SELECT dev.device_id, dev.name, dev.type, dev.connected, dev.on_state FROM rooms JOIN (SELECT * FROM deviceInfo JOIN deviceStatus WHERE deviceInfo.device_id = deviceStatus.device_id) as dev WHERE rooms.room_id = dev.room_id and rooms.name=? and dev.ipaddr=?`

    updateDeviceStatus string = `UPDATE deviceStatus SET connected=?, on_state=? WHERE device_id=?`

    deleteRoom string = `DELETE FROM rooms WHERE name=?`
    deleteDevice string = `DELETE FROM deviceInfo WHERE ipaddr IN (
        SELECT ipaddr FROM rooms JOIN deviceInfo ON rooms.room_id=deviceInfo.room_id WHERE rooms.name=? and deviceInfo.ipaddr=?
        )
    `
)

After that, we create the corresponding go functions to execute the SQL commands.

For example, create a AddRoom function below, we have a receiver parameter denoted by the (s *DatabaseManager) which allows one to associate a particular function, AddRoom, with a specific type, in this case, the DatabaseManager.

func (s *DatabaseManager) AddRoom(room models.RoomInfo) error {
    _, err := s.Db.Exec(insertNewRoom, room.Name)

    if err != nil {
        log.Println("insertNewRoom query failed")
        return err
    }

    return nil
}

We call s.Db.Exec to execute our insertNewRoom command and also the name of our room. We can then check if the insertion is successful by checking if the err is nil.

To query our database, we can call .Query, which will return to us all the rows that satisfy our conditions. We need to call rows.Close() after reading so that we prevent memory leaks. We can read through all the rows by looping through rows.Next() and then call rows.Scan(<column names queries>). Below is an example for querying all the rooms we have.

func (s *DatabaseManager) GetRooms() ([]models.RoomInfo, error) {
    rows, err := s.Db.Query(getRooms)
    if err != nil {
        log.Println("getRooms query failed")
        return nil, err
    }
    defer rows.Close()

    rooms := []models.RoomInfo{}
    for rows.Next() {
        ri := models.RoomInfo{}
        err = rows.Scan(&ri.Name, &ri.Count)
        if err != nil {
            log.Println("failed to scan db rows")
            return nil, err
        }
        rooms = append(rooms, ri)
    }

    return rooms, nil
}

testing our application

To test the code that we had just written, let's write a function to populate our database:

// database.go

// for testing purposes
func PopulateDatabase() {

    Dbman.AddRoom(models.RoomInfo{Name: "livingroom"})
    Dbman.AddRoom(models.RoomInfo{Name: "bedroom1"})
    Dbman.AddDevice(
        models.RegisteredDevice{Name: "andre", Type: "wled", Ipaddr: "192.168.1.1", Hostname: "123"},
        models.DeviceStatus{Connected: false, On_state: false},
        "bedroom1",
    )
    Dbman.AddDevice(
        models.RegisteredDevice{Name: "betty", Type: "switch", Ipaddr: "192.168.1.2", Hostname: "123"},
        models.DeviceStatus{Connected: false, On_state: false},
        "bedroom1",
    )
    Dbman.AddDevice(
        models.RegisteredDevice{Name: "cathy", Type: "wled", Ipaddr: "192.168.1.3", Hostname: "123"},
        models.DeviceStatus{Connected: false, On_state: false},
        "bedroom1",
    )
    Dbman.AddDevice(
        models.RegisteredDevice{Name: "derick", Type: "switch", Ipaddr: "192.168.1.4", Hostname: "123"},
        models.DeviceStatus{Connected: false, On_state: false},
        "livingroom",
    )
    Dbman.AddDevice(
        models.RegisteredDevice{Name: "eagle", Type: "wled", Ipaddr: "192.168.1.5", Hostname: "123"},
        models.DeviceStatus{Connected: false, On_state: false},
        "livingroom",
    )
}

In our main, we can then initialize our database and call PopulateDatabase.

func main() {
    db := database.InitDatabase()
    database.InitializeGlobals(db)

    database.PopulateDatabase()

    //...
}

After running our main.go once, we can look into our database and check if it is correctly populated.

$ sqlite3 iothome.db
sqlite3> .table
// deviceInfo deviceStatus rooms

sqlite3> select * from rooms;
// 1|livingroom
// 2|bedroom1

sqlite3> select * from deviceInfo;
// room_id|device_id|name|ipaddr|type|hostname|macaddress
// 2|1|andre|192.168.1.1|wled||
// 2|2|betty|192.168.1.2|switch||
// 2|3|cathy|192.168.1.3|wled||
// 1|4|derick|192.168.1.4|switch||
// 1|5|eagle|192.168.1.5|wled||

sqlite3> select * from deviceStatus;
// device_status_id|device_id|connected|on_state
// 1|1|0|0
// 2|2|0|0
// 3|3|0|0
// 4|4|0|0
// 5|5|0|0

sqlite3> .exit

In SQLite3, the boolean is represented by integers. A 0 means false and 1 means true.

In the next part, we will use our DatabaseManager in our Gin routes and merge them.

Below is the full implementation of all our database receiver functions for the DatabaseManager type.

// database.go

func (s *DatabaseManager) AddRoom(room models.RoomInfo) error {
    _, err := s.Db.Exec(insertNewRoom, room.Name)

    if err != nil {
        log.Println("insertNewRoom query failed")
        return err
    }

    return nil
}

func (s *DatabaseManager) DelRoom(roomname string) error {
    _, err := s.Db.Exec(deleteRoom, roomname)

    if err != nil {
        log.Println("deleteRoom query failed")
        return err
    }

    return nil
}

func (s *DatabaseManager) DelDevice(roomname string, ipAddr string) error {
    _, err := s.Db.Exec(deleteDevice, roomname, ipAddr)

    if err != nil {
        log.Println("deleteDevice query failed")
        return err
    }

    return nil
}

func (s *DatabaseManager) AddDevice(dev models.RegisteredDevice, devStatus models.DeviceStatus, roomName string) error {
    row := s.Db.QueryRow(getRoomId, roomName)
    var roomId int
    err := row.Scan(&roomId)
    if err == sql.ErrNoRows {
        return err
    } else if err != nil {
        log.Println("getroomId query failed")
        return err
    }

    res, err := s.Db.Exec(insertNewDevInfo, roomId, dev.Name, dev.Ipaddr, dev.Type)

    if err != nil {
        log.Println("inserting entry into deviceInfo table failed")
        return err
    }

    var devId int64
    devId, _ = res.LastInsertId()

    _, err = s.Db.Exec(insertNewDevStatus, devId, devStatus.Connected, devStatus.On_state)

    if err != nil {
        log.Println("inserting entry into deviceStatus table")
        return err
    }

    return nil
}

func (s *DatabaseManager) UpdateDevStatus(roomName string, ipAddr string, devStatus models.DeviceStatus) error {
    device_id, _, _, err := Dbman.GetDevice(roomName, ipAddr)
    if err != nil {
        return err
    }

    _, err = s.Db.Exec(updateDeviceStatus, devStatus.Connected, devStatus.On_state, device_id)
    if err != nil {
        return err
    }

    return nil
}

func (s *DatabaseManager) GetDevice(roomName string, ipAddr string) (int, models.RegisteredDevice, models.DeviceStatus, error) {
    row := s.Db.QueryRow(getDeviceInfo, roomName, ipAddr)

    devInfo := models.RegisteredDevice{}
    devStatus := models.DeviceStatus{}
    var device_id int
    err := row.Scan(&device_id, &devInfo.Name, &devInfo.Type, &devStatus.Connected, &devStatus.On_state)
    if err == sql.ErrNoRows {
        return device_id, devInfo, devStatus, err
    } else if err != nil {
        log.Println("failed to scan db rows")
        return device_id, devInfo, devStatus, err
    }

    devInfo.Ipaddr = ipAddr

    return device_id, devInfo, devStatus, nil
}

func (s *DatabaseManager) GetDevices(roomName string) ([]models.RegisteredDevice, map[string]models.DeviceStatus, error) {
    rows, err := s.Db.Query(getDeviceInfoByRoom, roomName)
    if err != nil {
        log.Println("getDeviceInfoByRoom query failed") 
        return nil, nil, err
    }
    defer rows.Close()

    devList := []models.RegisteredDevice{}
    devStatus := make(map[string]models.DeviceStatus)
    for rows.Next() {
        rd := models.RegisteredDevice{}
        ds := models.DeviceStatus{}
        err = rows.Scan(&rd.Name, &rd.Ipaddr, &rd.Type, &ds.Connected, &ds.On_state)
        if err != nil {
            log.Println("failed to scan db rows")
            return nil, nil, err
        }

        devList = append(devList, rd)
        devStatus[rd.Ipaddr] = ds
    }

    return devList, devStatus, nil
}

func (s *DatabaseManager) GetRooms() ([]models.RoomInfo, error) {
    rows, err := s.Db.Query(getRooms)
    if err != nil {
        log.Println("getRooms query failed")
        return nil, err
    }
    defer rows.Close()

    rooms := []models.RoomInfo{}
    for rows.Next() {

        ri := models.RoomInfo{}
        err = rows.Scan(&ri.Name, &ri.Count)
        if err != nil {
            log.Println("failed to scan db rows")
            return nil, err
        }

        rooms = append(rooms, ri)
    }

    return rooms, nil
}

Golang can be installed through the official website. I will be using Go version 1.19.

Once Go is installed, we can set up our backend project structure.

Create our backend folder and initialize our backend workspace with go mod init. This helps to track our Go dependencies.

mkdir backend
cd backend

# module path can be anything but best is to be the same path as repo
go mod init <module path>

creating the gin engine

Gin is a web framework that we can create API routes for our front end to interface with. To start, we can first import the module into our Go project.

go get -u github.com/gin-gonic/gin

Then inside our router.go file, create a InitRouter function to initialize our router instance as r.

The router object r is then used to define various HTTP routes for the API. It takes two arguments, the first argument is the HTTP route in string and the second argument is the handler function that will perform a specific action to process the HTTP request. For example, r.GET("/", getServerStatus) maps a GET request to the root URL path to a handler function called getServerStatus.

HTTP methods used here include GET, POST, and DELETE. GET requests are commonly used to retrieve information from the server and are considered safe, as multiple calls to the server do not change its state. Therefore, HTTP GET requests are considered idempotent. In contrast, HTTP POST requests are used to create or update data on the server, potentially altering the server's state and making them non-idempotent. We should keep in mind this when implementing our handler function to ensure this holds and to use each HTTP method appropriately depending on the nature of the request.

The summary of the code in the router.go file is shown below:

// router.go

package routes

import (
    "github.com/gin-gonic/gin"
)

func InitRouter() *gin.Engine {
    r := gin.Default()

    r.GET("/", getServerStatus)

    r.POST("/create-room", createRoom)
    r.GET("/rooms", getRooms)

    return r
}

Overall, this code sets up a web server that listens for HTTP requests and routes them to the appropriate handler functions to process the requests and return responses.

In our main.go file, we can then call our InitRouter function and then start the server by calling r.run(). Using a function with names that start with an uppercase letter will cause it to be visible to other packages in your workspace. A function starting with a lowercase letter can only be used within its own package and cannot be called from another package.

// main.go

package main

import (
    // path to my routes folder
    "github.com/AndreWongZH/iothome/routes"
)

func main() {
    r := routes.InitRouter()

    r.Run(":3001")
}

creating handler functions

Let's first implement the HTTP / route's handler function so that we can query if our server is online. All the handler functions take *gin.Content as a parameter. Calling ctx.JSON here will send a JSON response back to the client. If our backend server is online, we will receive a JSON response of {"status": "ok"} back.

// routes.go
package routes

func getServerStatus(ctx *gin.Context) {
    ctx.JSON(http.StatusOK, gin.H{
        "status": "ok",
    })
}

We will then create two other handlers, createRoom and getRooms that will update our internal list rooms holding all the room names. For our POST route to /create-room, we want to pass the name of our room as a JSON payload in our HTTP request. We will then create a RoomJSON struct which can hold a collection of fields. Our RoomJSON currently has a field called Name that is of type string. The struct tag is denoted by the two backticks and within that contains json:"name". This helps to control the encoding of JSON and will use the lowercase version of name rather than the field Name. We can then call ctx.BindJSON(), which will bind the JSON payload to our RoomJSON struct. If our operation is successful, we can then append the room name to our list of room names.

Lastly, create our getRooms function that will return our list of rooms as JSON data and can see the rooms that we have added previously.

// routes.go
// package routes

// hold our list of room names
var rooms []string;

// struct to hold JSON body data
type RoomJSON struct {
    Name string `json:"name"`
}

func createRoom(ctx *gin.Context) {
    var roomJSON RoomJSON
    err := ctx.BindJSON(&roomJSON)
    if err != nil {
        ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{
            "error":   "failed to parse JSON data",
        })
    }

    rooms = append(rooms, roomJSON.Name)

    ctx.JSON(http.StatusOK, gin.H{
        "status": "room name successfully added",
    })
}

func getRooms(ctx *gin.Context) {
    ctx.JSON(http.StatusOK, gin.H{
        "rooms": rooms,
    })
}

running and testing our server

we can run our backend server by running

go run main.go

and the sever will be setup as shown below

To test if our API endpoints are working, we can make an HTTP request to the server either on the command line or an API platform such as Postman.

Using command line

# test if server is online
$ curl http://localhost:3001
# will return {"status": "ok"}

# add a room name
$ curl -X POST -d '{"name": "room1"}' http://localhost:3001/create-room
# will return {"status":"room name successfully added"}

# view all rooms
$ curl http://localhost:3001/rooms
# will return {"rooms":["room1"]}

Using Postman

Check if the server is online

Adding a new room name

View all room names

Seems like all our endpoints are working fine.

However, the room names that we just stored are only persistent only when our server is running. If our server throws an error unexpectedly and exits, we will lose all our data. To make our storage persistent, we will introduce a database into our project.

In part 3, we will look at how to integrate the SQLite3 database into our application to make our data storage persistent. We will also add more handler functions to handle more request types.

IOT Home is an application I created to help manage my smart Home. The application allows me to control lights and switches in my house, providing me with a convenient and intuitive way to manage my home automation.

My inspiration for IOT Home came from Home Assistant, an open-source home automation software that can connect with many smart devices. I wanted to create a similar application that I could use daily, with a focus on controlling lights and switches. I also wanted to explore a variety of technologies and integrate them into my project.

Application Overview

IOT Home is a web application that provides a dashboard to control lights and switches in my home. The application's current focus is on controlling lights and switches, but it can be extended to control other smart home devices in the future.

Here's a preview of the application we will be building:

App features

Below is a list of features to build for my first minimal version of the app.

Authentication

• Users can log in to the dashboard

• Users can logout of the account

• Users can signup for an account

Rooms

• Users can create a room

• Users can view all the rooms created

• Users can delete rooms

Devices

• Users can add a device to a room

• Users can view all devices in a room

• Users can see the status of the device

  • check if the device is currently connected

  • check if the device is currently on

• Users can access the settings page of a device

• Users can switch a device on or off

• Users can delete a device

• Device type can support

  • WLED

  • Tasmota

Application Architecture

Below is a simple architecture diagram that shows the interaction between the different components of the application that I will be working on. The initial communication protocol between the device and the server will be through REST API, similar to the communication between the server and the front end. Our data will be persistent through the use of a database in which we can insert and modify the data through SQL queries.

Tech stack

Some of the main technologies I will be using are listed below:

• Next.js

• Golang

• Sqlite3

Next.js

Next.js is a React-based web framework that I will be using to write my frontend part of the application. I like Next.js as they simplify development for my project. It uses a built-in router that automatically generates routes based on the file system structure of the project. Next.js also uses TypeScript which helps in static type checking in JavaScript code.

Golang

I plan to use Golang as my backend server programming language due to its simplicity, efficiency and concurrency support. It is a popular language for building backend services.

SQLite3

SQLite3 is a lightweight relational database management system that is used in small-scale applications. I believe that utilizing SQLite3 will provide me with an opportunity to enhance my knowledge about relational databases and their optimization techniques.

I don't think this is a perfect guide to creating an application but more of a documentation of my thought process and difficulties when developing an application from scratch.

Check out part 2 on creating a backend server with Golang and Gin