x86 Tiny ESXi Lab Build
The home lab
As a systems engineer for Extreme Networks, I like to get as much hands-on lab gear that I can within a reasonable budget. I have quite a large lab setup at home as you can see.
One of my goals was to build something a bit more portable and powerful enough to run ESXi with a few VMs. I also like things that don’t take up too much power sitting idle. My test lab configurations usually consist of different virtual network operating systems such as Extreme Networks EXOS as well as Extreme virtual Wi-Fi controllers, Extreme Control VMs, and a host of other VMs. I usually don’t generate large amounts of traffic or massive compute load in my lab. If I need more computing power, I move to my physical switches and higher end Sandy-bridge based XEON servers.
The x86 Based ODROID H2
After a bit of hunting, I found a nice and small Supermicro rig, the SYS-E200. You can check out an awesome portable rig built using a few SYS-E200 units at tinkertry.com here. One of the downsides is that the SYS-E200 build per unit can be quite expensive, so I started to look for a smaller x86 system on chip (SoC) solution. SoC tends to be cheaper, draw less power, but the downside is they aren’t typically powerful. I then came across the x86 SoC Odriod-H2 system by Hardkernel which sported a Gemini Lake Intel CPU with VT-x virtualization support. This board looked like a perfect small and portable solution to run ESXi on. I quickly placed a pre-order as the board started at just $112. Here are the full specifications:
- Intel Quad-core processor J4105 (14nm) with 4MiB Cache, up to 2.5Ghz(Single Thread) or 2.3Ghz(Multi Thread)
- Dual-channel Memory DDR4-PC19200 (2400MT/s)
- Total 32GiB RAM Space with two SO-DIMM slots
- 4 x PCIe 2.0 for one M.2 NVMe storage
- 2 x Gbit Ethernet ports
- 2 x SATA 3.0
- SSE4.2 accelerator (SMM, FPU, NX, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AES)
- Intel UHD Graphics 600 (Gen9.5 LP GT1) up to 700Mhz
- HDMI 2.0 and DP 1.2 multiple 4K/60Hz video outputs
This tiny x86 lab box has some impressive specifications. I didn’t need lots of CPU power, but with dual SATA ports, 32GB max RAM, M.2 support, VT-x, and 2 Gigabit interfaces I couldn’t pass this board up. I’m glad I preordered because the units sold out pretty quick. Here are the total build costs so far:
|ODRIOD-H2 Board (ameridroid.com)||$112|
|ODRIOD-H2 Type 3 Case (ameridroid.com)||$12|
|ODRIOD-H2 Power supply (ameridroid.com)||$12|
|ODRIOD-H2 Power switch (ameridroid.com)||$8|
|SATA/DATA Power Cable (ameridroid.com)||$4|
|4GB RAM Patriot Signature 2400MHz PC4-19200 CL17 SODIMM (amazon.com)||$30|
|4GB RAM Samsung DDR4-2400MHz non-ECC Unbuffered CL17 SODIMM (amazon.com)||$31|
|Older 64GB SSD on hand||NA|
The power switch isn’t required as the board does have a small power and reset button that’s accessible through a small opening in the ODROID case, but I thought it would function a bit better with a larger power button. You could even run with NVMe or eMMC storage only and go with the smaller Type 2 case.
Once I received the board, I inserted the first 4GB RAM module (Patriot), and to my dismay, I couldn’t get the unit to post. Nothing I tried worked. I quickly posted to the ODRIOD forum and noticed other people were having issues with different RAM modules. I quickly ordered a second RAM stick that was on the officially supported list, and I finally got the board posted to the BIOS. I would have gone with a larger capacity module but didn’t want to spend the extra cash just in case the board was DOA. One thing for sure is that this unit is picky on RAM, so make sure you order from the official HardKernel supported list. From reading through the rest of the forums, a BIOS update is in the works to correct some of the RAM compatibility issues.
ODRIOD H2 and ESXi
The next task was to get ESXi installed. I did some preliminary research on the NICs which were Realtek RTL8111G units. I quickly found that newer versions of ESXi didn’t have these drivers baked in, so I started looking at how to add the drivers into ESXi. I followed an article from sysadminstories.com for the how-to. I started with ESXi 6.5.0 update 2 offline bundle. Free ESXi doesn’t have a 6.7 offline bundle that I could find.
Once you have a USB ESXi image ready, plug it in. Set the bios order to boot via USB and install ESXi. I decided to install a SATA SSD in my ODRIOD as I had an extra 64GB drive laying around which saved on the build cost. I also had to manually modify precheck.py during the ESXi installation since the system detected less than 4GB of RAM and my other RAM module wasn’t working. Here’s another article that shows the steps from simon-simonnaes.blogspot.com.
I then loaded up one of my favorite virtual network operating systems, Extreme Networks virtual_EXOS ISO from the Extreme Networks GitHub page. If you get the following error during the EXOS installation “mount: mounting /dev/hdc on /mnt/a failed: No such device or address” make sure your VM cd-rom drive is on IDE controller 1 and set to Master in Vmware. I added the first Realtek NIC assigned to vSwitch0 on port 2 of EXOS and the second Realtek NIC assigned to a new vSwitch to port 3 on EXOS. I’m now running a two-port virtual_EXOS system that bridges traffic across both ODROID-H2 NICs.
With three H2s, I could also try and build an HCI demo lab possibly running Nutanix CE. A FreeNAS build would also be a cool project. You’d have to check driver support, but as of the end of December 2018, the ODROID H2 is still on back order. Overall, this was a pretty fun build.