Proxmox Deploy

Description

Proxmox Deploy is a little script to manage my HomeLab with JSON file.

Why?

As my homelab was growing I realised that it was harder and harder to keep everything in sync and up to date. So I decided to create a script to manage my Proxmox homelab.

How it works

The concept is simple, you have a Git repository with a the following structure:

.
├── config.json
├── lxc
│   ├── <id>
│   │   ├── config.json
│   │   ├── <your files>
│   │   └── <your folders>
│   └── <id>
│       ├── ...
│── qemu
│   ├── <id>
│   │   ├── config.json
│   │   ├── <your files>
│   │   └── <your folders>
│   └── <id>
│       ├── ...
│── scripts
│   ├── <your scripts>
│   └── ...

See below for more information about the structure and differents files

PDJ (Proxmox Deploy JSON) is a program that will read that repository files and execute the necessary commands to create/update your LXC/VM. Now ideally you have some sort of Git actions like Drone/GitHub(/Gitea) Actions to run PDJ automatically when you push a change, which will result in your homelab being updated almost instantly.

If you don't, no big deal, you'll just have to manually clone and update your repo, then start PDJ (you could use a crontab with regular intervals)

Usage

Download

Download the pre-compiled binaries from the release page or build it yourself.

Build it yourself

# Build on Debian
git clone <url of this repo>
cd ProxmoxDeploy
apt update && apt install -y build-essential patchelf
pip install nuitka
pip install -r requirements.txt
python -m nuitka --onefile run.py --output-filename="ProxmoxDeploy"

Also see the Dockerfile and .drone.yml for more information.

Run it

# Run it
./ProxmoxDeploy --repo /path/to/repo

Documentation

Configuration

General

Before configuring your LXC and VM, you must decide how you'll run this program. As it requires SSH for some actions, you have two options:

• Run it directly on the Proxmox VE host (recommended) (refered as local)
  • No configuration needed
  • You can use Git(ea)/Drone actions to run it automatically as soon as a change is pushed.
• Run it on another machine and connect via SSH
  • You'll need to setup passwordless SSH connection between your machine and the Proxmox VE host.

Proxmox VE

The Proxmox VE configuration is located in the config.json file.

{
  "pve":{
    "host": "<pve host ip/hostname>",
    "user": "<user to connect via ssh>", // usually root
    "port": 22, // ssh port 
    "local": false // set to true if this program is running directly on the PVE without needing ssh
  }
}

LXC

To create a LXC, you need to create a JSON file in the resources/lxc/<id of lxc> folder. You can look at the example with ID 100 in the resources/lxc/100 folder.

Here is an example config with all available parameters :
Note : this is not valid JSON, it's just for documentation purposes, if you want to copy the file, use the one in the resources folder

{
  "lxc_hostname": "traefik", //hostname of the lxc
  
  // the os block contains the os type
  // alpine | archlinux | centos | debian | devuan | fedora | gentoo | nixos | opensuse | ubuntu | unmanaged
  "os": {
    "name": "alpine",
    "release": "3.17"
  },
  
  // the resources block contains the resources given to the lxc
  "resources": {
    "cpu": "2", // number of cpu cores
    "memory": "1024", // memory (ram) in MB
    "swap": "256", // swap in MB
    "disk": "8", // disk size in GB
    "storage": "local-lvm" // the proxmox storage to use
  },
  
  // the network block contains the network configuration
  "network": {
    "bridge": "vmbr0", // the proxmox bridge to use, vmbr0 is the default one
    "ipv4": "dhcp", // ipv4 address, dhcp for dhcp, auto for auto, or an ip address
    "ipv6": "auto", // ipv6 address, dhcp for dhcp, auto for auto, or an ip address
    "mac": "92:A6:71:77:8E:D8", // mac address, leave empty for random
    "gateway4": "", // ipv4 gateway, leave empty for auto/dhcp
    "gateway6": "", // ipv6 gateway, leave empty for auto/dhcp
    "vlan": "" // vlan id, leave empty for no vlan
  },
  
  // the options block contains various options for the lxc
  "options": {
    "privileged": "false", // set to true to run the lxc in privileged mode
    "start_on_boot": "false", // set to true to start the lxc on boot
    "startup_order": 2, // the startup order of the lxc
    "password": "qwertz1234", // the password of the root user, leave empty for none
    "tags": "2-proxy+auth" // tags for the lxc to display in the proxmox web ui
  },

Tip : have a look at the man page of the pct command to see how the values should look.

Creation

In the creation section, you have the option to define conditions for checking whether your LXC/VM has been previously configured.
If all of these conditions are met, the script will proceed directly to the deploy section and execute the necessary steps to update it. (This indicates that your LXC/VM has already been configured, and we want to avoid erasing any existing settings/config.)
However, if any of the conditions fail to match, the script will process to the creation steps section and execute all the commands again.

Conditions

File

The file condition will check if a file exists or not inside the LXC/VM.

"conditions": {
  // use a single file
  "file": "/var/data/traefikv2/traefik.toml"
  // or use a list
  "file": ["/var/data/traefikv2/traefik.toml", "/var/data/config/traefikv2/docker-compose.yml"]
}

It can be an array of file using ["file1", "file2"] or just one file in double quote "file".

Folder

The folder condition will check if a folder exists or not inside the LXC/VM.

"conditions": {
  // use a single folder
  "folder": "/var/data/traefikv2"
  // or use a list
  "folder": ["/var/data/traefikv2", "/var/data/config/traefikv2"]
}

It can be an array of folders using ["folder1", "folder2"] or just one folder in double quote "file".

Programs

The programs condition will check if a program is installed or not in the LXC/VM.

"conditions": {
  // use a single program
  "program": "docker"
  // or use a list
  "program": ["docker", "docker-compose"]
}

It can be an array of programs using ["program1", "program2"] or just one program in double quote "program".
Note:This uses which to check if the program matches to anything

Command

The command condition will check if a command returns a specific value.

"conditions": {
  "command": "whoami",
  "value": "root"
}

Note: This uses bash -c to execute the command
Note: If value is an integer, it will look for the return code, if it's a string it's check the output.

Docker

The docker condition will check if a docker (or podman) container is running or not.

"conditions": {
  // use a single container
  "container": "traefikv2"
  // or use a list
  "container": ["traefikv2", "portainer"]
}

Steps

Note: Paths
When you have to specify a path, you can use /global/ to use the global folder (resources/<scripts|file>/).
If nothing is specified, the script will use the VM/LXC folder (resources/lxc/<id>/).

The comments in the JSON below are only for documentation purposes and are not valid JSON, remove them before running.

Script

The script step will execute a script.

"steps": [
  {
    "type": "script",
    // use path for local scripts
    "path": "global/install-docker.sh" // path in repo (here: resources/scripts/install-docker.sh)
    // or use url for remote scripts
    "url": "https://xyz.abc/scripts/install-docker.sh" // remote url
    // or use lxc_path for scripts inside the lxc
    "lxc_path": "/root/install-docker.sh" // lxc path
  }
]

File

The file step will create / copy a file to the VM/LXC.

"steps": [
  {
    "type": "file_create",
    "path": "/var/data/traefikv2/traefik.toml", // lxc/vm path
    "permissions": "644" // (optional) permissions of the file
  }
]

"steps": [
  {
    "type": "file_copy",
    "path": "traefik.toml", // local path (here: resources/lxc/<id>/traefik.toml)
    "destination": "/var/data/traefikv2/", // lxc/vm path
    "permissions": "644" // (optional) permissions of the file
  }
]

Note: file_copy creates the folder if it doesn't exist

Folder

The folder step will copy a folder to the VM/LXC.

"steps": [
  {
    "type": "folder_create",
    "path": "/var/data/traefikv2", // lxc/vm path
    "permissions": "755" // (optional) permissions of the folder
  }
]

"steps": [
  {
    "type": "folder_copy",
    "path": "data/", // local path (here: resources/lxc/<id>/data/)
    "destination": "/var/", // lxc/vm path
    "permissions": "755" // (optional) permissions of the folder
  }
]

Note: folder_copy creates the folder if it doesn't exist

Command

The command step will execute a command on the VM/LXC.

"steps": [
  {
    "type": "command",
    "command": "whoami", // command to execute
    "working_directory": "/var/data/config/traefikv2" // (optional) lxc/vm path
  }
]

Docker

The docker step will execute a command inside a docker container running on the VM/LXC.

"steps": [
  {
    "type": "docker",
    "container": "<container-name>", // docker container name
    "command": "<command-to-run-inside>" // docker command to execute
  }
]

Docker-compose

The docker-compose step will execute a docker-compose command on the VM/LXC.

"steps": [
  {
    "type": "docker-compose",
    "command": "up -d", // docker-compose command to execute
    "working_directory": "/var/data/config/traefikv2" // lxc/vm path
  }
]

Note : Here, why not use the command step? Docker Compose can sometimes be executed with docker-compose or docker compose, the script will take care of choosing the correct one.

Git

The git step will clone a git repo on the VM/LXC.

"steps": [
  {
    "type": "git",
    "url": "https://git.abc.xyz/abc/abc.git", // git url to clone
    "destination": "/root/" // lxc/vm path
  }
]

Note: At the moment, no authentication is supported, so only works on public repo.

Download

The download step will download a file on the VM/LXC.

"steps": [
  {
    "type": "download",
    // download a single file
    "url": "https://git.abc.xyz/file.tar.gz", // download url
    // or use a list of urls
    "url": ["https://git.abc.xyz/file1.tar.gz", "https://git.abc.xyz/file2.tar.gz"] // download urls
    "destination": "/tmp/" // lxc/vm path
  }
]

Extract archive (tar/zip)

The unzip step will unzip a file in the VM/LXC.

"steps": [
  {
    "type": "unzip",
    "path": "/tmp/file.tar.gz", // lxc/vm path to the archive
    "destination": "/var/data/config/traefikv2" // (optional) lxc/vm path to extract the archive, will use archive parent directory if blank
  }
]

Note : At the the moment, only tar and zip are supported.

Install package

The install-package step will install a package on the VM/LXC.

"steps": [
  {
    "type": "install-package",
    // install a single package
    "package": "git",
    // or use a list of packages
    "package": ["git", "docker"]
  }
]

Note : At the the moment, only apt, apk, dnf, yum are supported.
Warning : Packages can have different names depending on the Linux distribution.

Remove package

The remove-package step will remove a package on the VM/LXC.

"steps": [
  {
    "type": "remove-package",
    // remove a single package
    "package": "git"
    // or use a list of packages
    "package": ["git", "docker"]
  }
]

Note : At the the moment, only apt, apk, dnf, yum are supported.
Warning : Packages can have different names depending on the Linux distribution.

Power

The reboot step will reboot the VM/LXC.

"steps": [
  {
    "type": "reboot"
  }
]

The start step will start the VM/LXC.

"steps": [
  {
    "type": "start"
  }
]

The stop step will stop the VM/LXC.

"steps": [
  {
    "type": "stop"
  }
]

Replace in file

The replace-in-file step will replace a string in a file.

"steps": [
  {
    "type": "replace-in-file",
    // replace in a single file
    "path": "/var/data/config/traefikv2/traefik.toml", // inside lxc/vm
    // or use a list of files
    "path": ["/var/data/config/traefikv2/traefik.toml", "/var/data/config/traefikv2/traefik2.toml"] // inside lxc/vm
    "search": "abc", // string to search
    "replace": "xyz", // string to replace
    "case_sensitive": true, // (optional) case sensitive? default: true
  }
]