Ceph Cluster Raspian (english version)

November 17, 2017
Raspian Raspberry PI Ceph

Raspian / Ceph

Here is all the good stuff:


We will not use CephFS, so we will not deploy mds components. You will find more informations in the official documentation: Doc

A best practice is to not install mon and osd on the same machine. But we only have 3 machines so we will not follow this advice. So we will have this architecture :

Hostname Function
Ceph01 Admin/Monitor/OSD
Ceph02 Monitor/OSD
Ceph03 Monitor/OSD

This setup is not the best but it will be enough for our tests. Please note that ceph01 have an admin function. This is due to the fact that we will deploy the cluster from this machine using ceph-deploy.

There are two very points :

The Stretch packages are too old ( version 0.94 soit Hammer : Ceph Release ).

The Ceph project doesn’t provide (not yet ??) packages for the armhf architecture. So we will grab them in testing.

Here are the traps :

* Install the good version of *libleveldb1v5* or else Ceph tools will not work
* Install *ceph-deploy* **before** *ceph* to avoid dependencies problems with python's packages.
* Install *btrfs-progs* to format OSD with *btrfs*. It's not mandatory, *XFS* is preferred
* Install *rbd-nbd* because the *rbd* module doesn't exist on Raspbian

This will allow the admin machine to connect on each other with ssh (even on itself).

From there, you should have a functionnal cluster but without OSD (so cluster’s health at HEALTH_ERR): bash $ ceph -s Now, we need to add OSD to our cluster. For it we will use our usb keys like this: * ceph01 : 2 keys ( /dev/sda and /dev/sdb ) * ceph02 : 2 keys ( /dev/sda and /dev/sdb ) * ceph03 : 1 key ( /dev/sda )

We will initialize our keys (still as ceph user): bash $ ceph-deploy disk zap ceph01:sda ceph01:sdb ceph02:sda ceph02:sdb ceph03:sda

One initialized, we will format them. I choose BTRFS, but it’s not mandatory. By default it will be XFS : bash $ ceph-deploy osd prepare --fs-type btrfs ceph01:sda ceph01:sdb ceph02:sda ceph02:sdb ceph03:sda

This command will create two partitions on each key. One for the data and one for the journal.

See http://btrfs.wiki.kernel.org for more information. Detected a SSD, turning off metadata duplication. Mkfs with -m dup if you want to force metadata duplication. Performing full device TRIM (1.00GiB) … Label: (null) UUID:
Node size: 16384 Sector size: 4096 Filesystem size: 3.00GiB Block group profiles: Data: single 8.00MiB Metadata: single 8.00MiB System: single 4.00MiB SSD detected: yes Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 3.00GiB /dev/nbd0 ```

Mount it and write:

# mount /dev/nbd0 /mnt && cd /mnt && echo test > test 

Docker swarm

January 3, 2018
Raspian Raspberry PI Docker

Docker on Raspbian

January 3, 2018
Raspbian Raspberry PI Docker

Cluster Ceph Raspian

November 17, 2017
Raspian Raspberry PI Ceph