← Back to Course Index

Week 10 Advanced Guide: Ceph Cluster Setup

Module: Operating Systems 3 (Virtualisation & Cloud Technologies) Instructor: KT Nshimba
Level: Advanced / Optional
Purpose: Reference guide for building a Ceph Storage Cluster.

1. Architecture Overview

A functional Ceph cluster consists of three types of daemons:

1. Monitors (MONs): Maintan the cluster map state. You need at least 3 for High Availability (Quorum).
2. Managers (MGRs): Keep track of runtime metrics and current state (dashboard, orchestration).
3. Object Storage Daemons (OSDs): The workhorses. One OSD daemon per physical hard drive. They store the actual data.

2. Prerequisites

To build a production-grade cluster (or a robust lab), you need:

• 3 Nodes (Servers): e.g., ceph-node-01, ceph-node-02, ceph-node-03.
• Networking:
  • Public Network: For client traffic (OpenStack Cinder talks here).
  • Cluster Network (Optional but recommended): Private backend network for replication traffic.
• Raw Disks: Unformatted hard drives (e.g., /dev/sdb, /dev/sdc) on each node. Do not format them. Ceph manages the raw partitions directly.

3. Installation Strategy: Cephadm

Modern Ceph deployments usage cephadm. It uses containers (Podman or Docker) to deploy the daemons, keeping the host OS clean.

Step 3.1: Bootstrap the First Node

On ceph-node-01:

# 1. Install cephadm
sudo apt update
sudo apt install -y cephadm

# 2. Bootstrap the cluster (Create the first Monitor and Manager)
# Replace <MON_IP> with the node's IP
sudo cephadm bootstrap --mon-ip <MON_IP>

• Result: This command pulls the Ceph container image, starts a Monitor and Manager, and generates a ceph.conf and keyring.
• Vital: It prints the Dashboard URL, User (admin), and Password. Save these!

Step 3.2: Add Hosts

You need to tell the cluster about the other nodes. Cephadm uses SSH to manage them.

1. Copy SSH ID: The bootstrap process created a public key (/etc/ceph/ceph.pub). Copy this to the other nodes. bash ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node-02 ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node-03

2. Add to Cluster: bash sudo ceph orch host add ceph-node-02 sudo ceph orch host add ceph-node-03

Step 3.3: Add Storage (OSDs)

Now tell Ceph to consume the raw disks.

Option A: One-by-One (Manual)

sudo ceph orch daemon add osd ceph-node-01:/dev/sdb
sudo ceph orch daemon add osd ceph-node-02:/dev/sdb

Option B: The "All Available" Spec (Automatic) This command tells Ceph: "Look at all nodes, find all empty disks, and turn them into OSDs."

sudo ceph orch apply osd --all-available-devices

4. Integration with OpenStack Cinder

Once your cluster is healthy (ceph -s shows HEALTH_OK), you are ready to connect it to Cinder.

1. Create a Pool: bash ceph osd pool create volumes 128
2. Initialize it for RBD: bash rbd pool init volumes

3. Create a User for Cinder: bash ceph auth get-or-create client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes' mgr 'profile rbd pool=volumes'

   • This outputs the Keyring data. Save this to a file: ceph.client.cinder.keyring.
   • Deploy: Copy this keyring to your OpenStack Controller as described in the Week 10 Student Notes (Section 2.3.3).

5. Summary Checklist

• Nodes prepared with Docker/Podman and Python3.
• cephadm bootstrap completed on Node 1.
• SSH keys distributed to Node 2 and Node 3.
• Hosts added via ceph orch host add.
• OSDs provisioned on raw disks.
• Pool volumes created and client.cinder user authorized.

← Back to Course Index