Posted on: 2018-03-18
When I started working as a network administrator in 2007 I got tasked to set up a new HP ProLiant server.
The server shipped with Windows 2003. I'm not really a Windows guy, but with just some clicking around I managed to set up an NT domain with server stored profiles for all the clients. But it crashed every other day. No custom software was installed on it - RAM was OK, drivers up-to-date, hardware diagnostics went fine, it was just really unstable. So during one weekend I backed up all the data, wiped the thing and put Linux on it. Since then it ran smoothly. (Later on we needed Windows 2003 for some other software, I put it in a VM, and it ran stable).
Problems started when we had to put more services on it. The thing had two 72GB, 2.5 inch SAS disks inside - in a proprietary enclosure. Expanding the storage was really expensive because you had to buy "genuine" HP parts which contained the same harddisks you could buy normally, only in an overpriced enclosure. At one point we had to upgrade the storage really really quickly so I went to the nearest supermarket which also sold some electronic products and bought three external USB harddisks. But I didn't just plug them in, I disassembled them and plugged the 2.5 inch SATA disks which were inside directly into the SmartArray bay. Of course the disks were just dangling off the SAS ports because they lacked the custom enclosure, but it worked quite well in the RAID 5 configuration. We had this system for many years ... (Did I mention this was quite a while ago? I wouldn't do this nowadays …) And by working "well" I mean "well" for HP standards. Writing to the disks was not only unbearably slow, it also blocked the rest of the disk subsystem! So I had to keep all the "live" data on the small SAS drives and move infrequently accessed data and nightly backups over to the RAID 5 :/ That took quite a bit of juggling ...
I wowed to never touch HP servers again.
Fast forward 8 years, at my new job we have only one HP box, a neat little MicroServer sitting quietly in a network cabinet. It has regular 3.5 inch SATA harddisks inside and works quite well. Even when we put FreeBSD on it, it had no issues with the hardware whatsoever. Except that it takes ages to boot. No servers boot fast, but HP boxes seem to take extra long.
Being happy about the HP server box I thought: "Hmm, maybe they have improved over the years..." So I bought two more, one for a client, one for our coworking space.
Then problems started rolling in:
- We got a bigger external backup disk for one of the servers - and with the attached USB disk it just wouldn't boot. A smaller USB disk works, but the bigger one crashes the bootloader. It's a known BIOS bug, even though we're not booting off that disk it messes up something in the INT 13h routines. So we had to stick with the smaller one - or always unplug the disk when booting the system, not really an option.
- One of the other servers just stopped recognizing harddisks in the first two bays. They suddenly disappeared. I put the disks in slot 3 and 4 and pray that they don't vanish as well.
The most issues I had with the third server:
- I put an SSD in the optical drive slot on the top. A quite common configuration, as I read online - but unsupported. I didn't know that. As soon as I started putting harddisks in the bays the system refused to boot of the SSD. All BIOSes I've come across allow you to change the boot priority, not HP, they know better of course.
- After settings the disks from AHCI to legacy it managed to boot of the SSD again, but then the network interface went missing. Doesn't show up in "ifconfig -a", "lspci" or anywhere else. Gone. HPs internal diagnostics show all system components as OK, but don't even mention a network interface. "Testing a network card? On a server? Why would anyone want to do that?". iLO works fine though, so it can't be a hardware issue.
- OK, let's fix the previous issues with a BIOS update. Jokes on me again, you can't just, like, get, a BIOS update from HP! To get a BIOS update you need a support contract or warranty or something. So I tried to set that up. Nope, won't recognize the serial number of this 4 months old device. Argh! Fuck it, let's just fetch the ISO image from some torrent site.
- Haha, the ISO is over 5GB - so you need a DVD burner and a blank double layer disc! The latter of which I don't have - Or, let's write it to an external USB disk ... Doesn't work of course. Different boot mechanism.
- Well, there's this iLO remote management thing which has support for remote media, let's try that Haha, you don't have iLO Advanced. iLO Standard can't do that.... Fuck it, there must be a keygen somewhere, let's just "register" this thing, clean up some shit left behind by HP - no way I'm paying for that - and remove the key again.
- ... If only the Remote Console would work! No, it's just a white window, not doing anything. No error or timeout, just sitting there doing nothing. ARRRRRGHHH!
- Again, Fuck this shit, let's just use a USB 3.0 Ethernet adapter. That works. for now ... Now I have a server with two NICs on board, none of them working.
Maybe I'll finally learn my lesson: NEVER BUY HP HARDWARE AGAIN!
The other non-HP servers we manage are mostly Supermicro boards and they work great! They have all kinds of hardware configurations like mixed SSDs and HDDs, additional 1G and 10G NICs... One even had an ISDN card. None of the issues mentioned above and the systems are rock solid! Of course the remote consoles have some quirkses - I haven't seen any remote KVM solution that doesn't suck - but at least it works. Also with Supermicro you get all features right out of the box, no additional license costs.
Why can't a big company like HP get things right?
The Dell servers we manage also have some weird proprietary disk systems, but at least you can get enclosures for a reasonable price.