[Cat: Site.performance] Issues present at one time of day once a week. (Part 1)

Right-o team, we have a problem.

My server is aging. I put it together from a piece-meal of parts sourced from the local auction website: Trademe.co.nz (similar to eBay, Amazon, Craigslist etc)

It’s got a::
Motherboard: Tyan Dual-Socket S7002
CPU: 2x Xeon X5560’s (2.8 GHZ)
OS HDD: 2x WD Velociraptor Harddrives (Raid 1)
Personal Storage: 5x 4TB WD-SE HDD’s
Onboard HDD Controller Storage HDD:
PCIE Raid Controller (IBM ServeRAID M5014 SAS): 5x WD 1TB Enterprise Class Yellow HDD”s
BMSworld VM storage: RAID10 (1+0 nested): approx 3TB

It’s been heavily modded from top to bottom.
The server chassis was originally a 1990’s Bank server – I paid 50$ for it from Trademe and had it shipped down from the north island of New Zealand.
It had a shitty cooling design of front to back. Front Active fan pressure, the back: passive.

This was the first change I made – by which I rotated the cooling design. Cold air in the bottom via 2x 180mm fans, warm air out the top via a series of holes I drilled as a vent. Works really really well and is damn quiet for a server like this.
The chassis also has the benefits of being¬†sectional.¬†Motherboard and I/O on one side, HDD’s and PSU’s on the other separated by a double wall.
The HDD’s that this chassis was designed for was the old SCSI 5inchHDD’s from the mid to late 90’s, but naturally that’s the same size as a CDROM bay. Hmm.
The only solution available was from ICYdock.com – take 3x drive bays insert enclosure and viola 5x 3.5′ HDD’s vertically. Add another 2 and that’s 15 drives. Awesome except I now need server grade HDD’s to suit being in a RAID array on the server… until then…