36 lines
3.4 KiB
HTML
36 lines
3.4 KiB
HTML
|
<!--
|
||
|
|
title: Soleil Levant
|
||
|
|
description:
|
||
|
|
published: true
|
||
|
|
date: 2024-06-23T15:03:01.932Z
|
||
|
|
tags:
|
||
|
|
editor: ckeditor
|
||
|
|
dateCreated: 2024-06-23T15:02:05.645Z
|
||
|
|
-->
|
||
|
|
|
||
|
|
<p><strong>DECOMISSIONED DUE TO RAID ARRAY FAILURE</strong></p>
|
||
|
|
<p>Soleil Levant was a server hosted in France for Project Segfault.</p>
|
||
|
|
<p>It was our most powerful server, and ran a large majority of our services.</p>
|
||
|
|
<figure class="image image_resized" style="width:302px;"><img src="https://wiki.projectsegfau.lt/images/thumb/4/43/Soleil_Picture.png/300px-Soleil_Picture.png" alt="" srcset="/images/thumb/4/43/Soleil_Picture.png/450px-Soleil_Picture.png 1.5x, /images/thumb/4/43/Soleil_Picture.png/600px-Soleil_Picture.png 2x" sizes="100vw" width="300"></figure>
|
||
|
|
<figure class="image image_resized" style="width:302px;"><img src="https://wiki.projectsegfau.lt/images/thumb/4/4e/Soleil_Open.jpg/300px-Soleil_Open.jpg" alt="" srcset="/images/thumb/4/4e/Soleil_Open.jpg/450px-Soleil_Open.jpg 1.5x, /images/thumb/4/4e/Soleil_Open.jpg/600px-Soleil_Open.jpg 2x" sizes="100vw" width="300"></figure>
|
||
|
|
<p>It ran Proxmox VE as its hypervisor.</p>
|
||
|
|
<h3>Decomissioning</h3>
|
||
|
|
<p>In mid-August, a disk on the server failed. This however was fixed after a 24 hour downtime (by getting the disk from someone the server host knew) and a backup restore for just our matrix instance which experienced slight database corruption.</p>
|
||
|
|
<p>After this, soleil ran relatively fine for a while, till 12/09/2023, when the server host noticed one of the drives were clicking again, and after investigation realized it went into a degraded state.</p>
|
||
|
|
<p>However, the spare drives we bought after the first disk failure didn't work since the LSI Hardware RAID was serialized to work with only hitachi drives (the drives we got had same part number and such but was from HP).</p>
|
||
|
|
<p>Later, when we got another hitachi drive from the same person, RAID failed to re build since 2 more drives failed. At this point, we thought we would just delete everything and restore from backup on a new ZFS, but then the LSI HBA we had didn't support IT firmware (so we basically cant disable raid and use the drives separately). This was the final nail in the coffin, and soleil levant was officially shutdown on 16/09/2023 due to these issues.</p>
|
||
|
|
<p>We however, didn't have any good server in the EU, so we had to move everything to our India server.</p>
|
||
|
|
<p>Along with that, we had to discontinue matrix which was too big.</p>
|
||
|
|
<blockquote>
|
||
|
|
<p>change of plans, i'll just fully retire Soleil Levant. it fucks me up mentally and financially and i don't feel like spending another sleepless night trying to get the server up when the most likely situation is that the entire RAID array is fucked and i'd need to replace every single disk + get an HBA for ZFS since i cant flash the raid card.</p>
|
||
|
|
<p>i know it's not the outcome you guys wanted but i need to focus on my health, sorry again</p>
|
||
|
|
<p>we'll make a blog post when everybody will be online and everything we want to back up is backed up but yeah.</p>
|
||
|
|
</blockquote>
|
||
|
|
<p>Message from MrLeRien (the server host) on our status page on 16/09/2023, when the server was discontinued</p>
|
||
|
|
<h3>Hardware</h3>
|
||
|
|
<p>The server was a Hitachi Compute Rack 210H, with dual socket Intel Xeon E5-2690s.</p>
|
||
|
|
<p>It has 320 GiB of memory, and a raid 5 array with 4 TB of usable space.</p>
|
||
|
|
<h3>VMs</h3>
|
||
|
|
<p>As with using Proxmox, the virtualization backend was QEMU/KVM.</p>
|
||
|
|
<p>The server was almost entirely virtualized, avoiding running anything on bare metal.</p>
|