2013-09-23 00:27: Added information about ESXi 5.5 not including RTL8111E support
2013-09-23 22:23: Added a link to installing ESXi 5.5 with RTL8111E support

In my last post, I covered my journey of finding a small, quiet, low-power virtualisation lab computer that can run nested virtualisation.

The final system

To summarise, here’s the system I ended up with, after much consideration:

  • Gigabyte Brix, Core i5 barebones kit (GB-XM11-3337) [Buy in the UK]
  • 16GB (2x8GB) SO-DIMM Crucial RAM
  • Plextor M5M 256GB SSD [Buy in the UK]
brix vs xbox controller

Total cost was roughly £575 ($920), but if you opt for a more modest Core i3 with 8GB RAM and a 128GB SSD you can make it as little as £370 ($600), which is much more palatable. Here’s a rough shopping list.

Power consumption idles around 12 watts running VMware ESXi 5.1 and I’ve seen it go up to 25 watts when I was loading it pretty hard with Turbo Boost at full pelt and the SSD being written to constantly.

I chose a higher-spec, more expensive system than I normally would, as I wanted to run an entire Nested Virtualization lab inside it including:

  • Multiple ESXi hypervisors, to emulate a cluster
  • Virtual router and virtual NAS to provide networking capabilities and shared storage inside the nested lab.
  • Multiple “Infrastructure” Virtual machines such as Active Directory, vCenter and vCloud
  • Multiple Hyper-V hypervisors to play with Microsoft’s offerings

Read more on nested virtualisation.

The Brix GB-XM11-3337 I opted for has an Intel Core i5-3337U 1.8GHz processor with Turbo Boost which supports both Intel VT-x with EPT (Extended Page Tables) for Nested Virtualization and VT-d for directly passing through connected hardware to a VM running on the ESXi hypervisor (The Core i3 doesn’t support VT-d, so bear that in mind if you fancy it).

In addition, the Gigabit NIC is a Realtek RTL8111E which works out of the box with a standard VMware ESXi 5.1 install. For me, this makes it more attractive than the Intel NUC, which needs some hackery to get the driver installed. I’m not lazy; I’m efficient 😉

brix rear port connectivity

Putting it together

This is what I love about these barebones systems; they take minutes to put together. The RAM slots in exactly like a laptop, and the mSATA drive is easy to install. Gigabyte even include a screw for you to screw down the mSATA drive after installing it (just make sure to unscrew it before you install the mSATA drive!)

mSATA screw

Configuration

Now everything’s installed, let’s tweak the configuration before installing ESXi.

You’ll need:

  • A USB keyboard
  • Either a TV or Monitor capable of taking a HDMI source, or a DisplayPort monitor.

I’ve been struggling to get the Mini-DP port to play nicely with my Dell U2412m, so I’ve resorted to using my HDMI TV for configuration.

Update: I’ve since found that a HDMI to DVI converter works well with my monitor. Mini-DP is still troublesome.

BIOS settings

  • Enable Virtualization
  • If you want lower idle and standby power usage, enable Erp. This does mean you won’t be able to use Wake-on-LAN, however.

Upgrade the Plextor M5M SSD firmware

The SSD arrived with Firmware version 1.02. This version appeared to have a power usage issue, which was picked up in the Anandtech review, but their testing method wasn’t 100% sound due to not being able to accurately measure the 3.3v rail. Either way, version 1.03 says it fixes a power consumption issue, so let’s upgrade the SSD.

As the Brix doesn’t have a DVD drive, we need to convert the ISO provided by Plextor to a bootable USB stick. Fortunately, Unetbootin does a great job of this:

  1. Download the latest version of the firmware in ISO format
  2. Download Unetbootin
  3. Write the ISO to a USB stick using Unetbootin
  4. Boot the Brix from the USB stick, and at the Unetbootin boot menu, choose the second boot option (the one immediately below Default).
  5. Follow the instructions to update the firmware. It’s very quick.

Installing ESXi

This is trivial, as all the components in the Gigabyte Brix are compatible with VMware ESXi 5.1 (woohoo):

  1. First you need to get the ESXi 5.1 ISO. You can get the Free Edition from VMware.
  2. Download Unetbootin, if you don’t have it already.
  3. Use Unetbootin and a spare USB stick to create a bootable USB stick from the ISO you just downloaded. Anything bigger than 1GB is fine.
  4. Boot the Brix from the USB stick
  5. Install ESXi

Note: If you want to install ESXi to the same USB stick you booted from, you can. The install files are in RAM, so you won’t saw off the branch you’re standing on 🙂 Running ESXi on a USB stick is perfectly fine in a lab and as far as I’m aware won’t degrade performance while it’s running as the OS is loaded into RAM.  If you don’t want to run it off the USB stick (it does look a bit untidy), install ESXi to the SSD.

That’s it! Easy-peasy 🙂

Brix running ESXi

My next post will cover Enabling and setting up Nested Hypervisors on ESXi

Fancy Hyper-V instead of ESXi as your base hypervisor?

If you want to install Hyper-V on your Brix as your base hypervisor, check out Jeff Hicks’ articles: Mini Hyper-V: Setup

Written by Phil Wiffen

Phil is an IT Professional working in Cambridge, England. He generally blogs about useful solutions that he comes across in his work/play.

19 Comments

Jim

Hi Phil. Have you tested GPU Acceleration in VMs? Since vmware documents says they extended vSGA to Intel and AMD GPUs. Thanks.

Phil Wiffen

Hi Jim. No, I have not tested GPU acceleration (haven’t had a need to). Where can I find the document you’re referring to?

Dave

Hi Phil. Are you having to press F10 at boot (assuming you installed onto the mSata SSD) to get the NUC to boot to the SSD? I am trying to get the system to boot without having to press F10 (in case of power outage).
Cheers,
Dave

Phil Wiffen

Hi Dave, currently mine boots from a USB stick, and I’m not seeing the same issue as you. I had to tweak the boot order in the BIOS, but that’s about it. I recently upgraded it to BIOS F6, in case that helps?

Hoss

Great article! Thanks for the write-up. I’m running my home “lab” on a HP N40L right now. I have it completely maxed out and I always push the limits with testing things out. I’ve been reviewing the Intel NUCs and Gigabyte BRIX, and I’m even contemplating waiting for AMD’s version of a NUC, but who knows when that will happen.

For now I’ve resorted to upgrading my desktop with 32GB and Workstation 10 and have a nested ESXi cluster running smoothly with the VMware Tools for Nested ESXi and VFRC mapped to small VMDK’s sitting on my desktops SSD. I moved the 16GB from my desktop in to my HTPC in the living room on a F2A85M-PRO/A10-5800K and got ESXi 5.5 installed by building a custom ISO.

My question to you is: how has the BRIX held up to all of your testing? Have you run in to any snafus that have stopped you from utilizing this little beast in a nested configuration with Hyper-V or ESXi?

Thanks!

Phil Wiffen

Hi Hoss,

It’s been very good at handling multiple VMs running concurrently; and it is super-quiet. My only issue is that I ended up needing to buy a NAS for VM storage as I ultimately ran out of space on the 256GB SSD; but that’s my own fault for not capacity planning properly 🙂

mururoa

Hi,
Great paper.
I just wonder why you choosed this model instead of a GB-BX model where you can install a 2″1/2 HDD together with the mSata SSD ? I think that having a 1 TB WD Red drive inside the box is a good thing.

John

Be careful here — the Haswell Gigabyte Brix (bx-i5-4200U) is a No-Go because of the missing VT-d and the stuck condition at “Initializing IOV” — the 3rd gen i5 seems to have VT-d as Phil’s install works. Weird that intel chose that path of having VT-x but not VT-d. I guess they needed to cut “somewhere”

I’ll go grab the new Gigabyte Brix Pro with the bxi5-4570R, the one that mururoa is referring to. The 4570R has the VT-d built in. I’ll confirm the ethernet support in esxi 5.5 and report back if it actually works. they have color choices too, so an High Availability strategy at home can be cool sounding (“Red side and black side” when describing it).

mururoa

Hi,
Well, it seems I was not careful enough and I did bought a 4200U 🙁
I was able to go past initializing IOV with noiommu option but then I ended stuck in setting (or initializing ) storage and was not able to go further.
Anyway since john mention the possible problem I were fast to test and so I was able to send back the Brix to the vendor.
And I ended with XM11 like yours 😉
I looked at brix pro it costs too much for my needs.
So I’ll have to keep some NAS to connect using USB 3 the 1 TB drive I bought for the 4200U. I prefer not to go in the nested storage using VT-D connected to virtualized filer even if it should be possible with VM startup order.

Sami

Hello.
The HM77 chipset motherboard in the GB-XM11-3337 doesn’t seem to support vt-d.

I am thinking of buying laptops (HP Pavilion 15-B140TX or HP ProBook 4540s) having HM77,HM76 chipsets respectively, and want virtualization capability.

The Intel ARK page above does not explicitly say that the chipset does NOT have “Vt-x” and “Vt-x EPT” support. Those features are entirely unmentioned.

Since you are using the HM77 motherboard, could you confirm if atleast “Vt-x” along with “Vt-x EPT” is available and working on it?

Phil Wiffen

Hi Sami, while I can’t speak for the laptops you mention, I’m fairly confident that the GB-XM11-3337 I have supports EPT and VT-d. I’m using 64-bit VMs with Nested Virtualization, and EPT is a requirement for that. I will try to take a screenshot over the weekend, but there’s more information on this on William Lam’s excellent site.

Madmini

Hi Phil,

I got the same setup as you. Working great for my homelab. Now I want to also use it a HTPC and for that I will need to be able to use the HDMI output. I have been trying to get GPU pass through working with no luck.

Have you tried this or know if this is possible?
Thanks

Phil Wiffen

Hi Madmini. I tried and it didn’t work for me; but that doesn’t mean it won’t work…more likely that I was doing something wrong. I have no real need to do it, so gave up very quickly.

Andrei Darie

Great Article, I follow all steps but I tried to install ESXi in Brix Model AMD A8-5545 http://www.gigabyte.com/products/product-page.aspx?pid=4748#ov. Is not Intel platform some you Use. In instalation procces I have one error when starts XORG process before instalation. I tried to see what kind problem have But is not posible find error log . You know how can see log error from instalation? I think if I can see the log maybe i can find one solution .
Thanks,
Andrei Darie

Andrei Darie

Hi , Finally I installed ESXi 5.5 I used Iso version with Network Adapter RTL8111 some you comment in this post and I added Sata AHCI driver because the default sata drivers from ISO is not compatible :(.

But finally i can say the following brix version is compatible with ESXi 5.5:
Brix AMD A8-5545 (4 core)
SSD Samsung EVO 250 Gb mSATA
8 Gb Kingstone DDR3 1333 Mhz

For me was very helpful your post.
Thanks !

Andrei Darie

Darren

Did you have any issues with tagged frames exiting the host to the physical switch? I cannot get any tagged frames in or out of the brix. I know my networking and its not working properly. I am wondering if anyone else has seen the same issue and is this the reason vmware dropped the realtek drivers.

Currently moving to 5.5 to see if anything changes.

Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *