Starting a homelab - Part 02 - Installing ESXi

First things first. Let's discuss the network diagram we will be working with:

As can be seen here, there is a L2/L3 managed switch between your home router and the rest of your infrastructure.
In my scenario, we will work with a few VLANs, so I will assume at this point that you are at least familiar with the concept of VLANs.

VLANs in this scenario:

• VLAN 2 - HomeNet (Device Management VLAN)
• VLAN 200 - Net1 - Business Network VLAN
• VLAN 210 - Net2 - DMZ Network VLAN

We will associate these VLANs with the following networks:

• VLAN 2 -> 172.28.0.0/24 (255.255.255.0)
• VLAN 200 -> 10.194.1.0/24 (255.255.255.0)
• VLAN 210 -> 10.195.1.0/24 (255.255.255.0)
  Of course, you can use whatever subnets you wish.

This setup allows you to access management interface of ESXi from your home network. Please keep in mind that this is not something that we would do in production networks and usually management VLAN is isolated from the rest of the network and only authorized personel have accesss to it. For simplicity though, in this setup we will use the same network as in your Home Network so that you can access it from anywhere - including your WiFi in case your Home Router is also a WiFi router.
L2 switch needs to be configured in the following way on ports towards the Home Router and the ESXi server:

interface GigabitEthernetX/0/X
 description WhiteBox ESXi MGMT
 switchport access vlan 2
 switchport mode access
end

This is a configuration example for Cisco switches, it may be different on your own device, and maybe it's even done through GUI (most likely, actually for non-enterprise appliances). With other vendors you will be looking for terms untagged instead of access and you will most likely assign VLAN 2 as untagged on port which connects to the ESXi. If you have a Unifi switch, look for something called switching profile and assign that profile to the port connecting to the server (under the assumption that you already configured it properly).

There is one more option, in case you only have one network interface on your server and still want to use trunking/VLANs on it. In that case, configuration will look something like this:

interface GigabitEthernetX/0/X
 description WhiteBox ESXi trunk
 switchport trunk native vlan 2
 switchport mode trunk
end

This configuration will allow ESXi server to utilize VLAN 2 as its management VLAN while still allowing all other VLANs to be tagged and pass through the trunk. Important note: VLAN IDs between your physical and ESXi infrastructure have to match! Meaning, if you're using VLAN 200 as your Business VLAN in your network, same ID needs to be used in your ESXi environment.

Why is this all necessary, you may ask yourself? One example would be if you want to virtualize your firewall. In that case, you will be able to pass all the traffic via trunk to the virtual firewall appliance, and then assign as many sub-interfaces (VLANs/networks) to it and point all your devices to use that firewall as its gateway. We will deal with this scenario in "ADVANCED: Virtualize your firewall and route traffic through it".

Let's go back to the setup with two network cards in your server for a moment. We still need to configure the second interface:

interface TenGigabitEthernetX/0/X
 description WhiteBox ESXi DATA
 switchport mode trunk
end

As you can see here, we are going to tag all VLANs on this interface and pass the tags to ESXi to deal with them.

Physical computers on the left side of the diagram are just as an example and they are assigned to their own respective networks - Net1 and Net2, so the configuration would be something like this:

interface GigabitEthernetX/0/X
 description Computer1-NET1
 switchport access vlan 200
 switchport mode access
end

interface GigabitEthernetX/0/X
 description Computer2-NET2
 switchport access vlan 210
 switchport mode access
end

Please understand that if you configure your ports like this that computers 1 & 2 will no longer be able to communicate, even if their IP addresses are in the same subnet. PSA: Computers assigned to the same network (in most scenarios/cases) need to be assigned to the same VLAN on L2 switch for them to successfully communicate.

With all of this out of the way, let's start with the install. Go to VMware and if you haven't already, register for an account there. After that, login to the user portal:

Once you are registered and logged in, choose All Products:

Under All Downloads select VMware vSphere Hypervisor (ESXi) and click on View Download Components.

Under Download VMware Sphere Hypervisor (ESXi) click on Download:

Under VMware Sphere Hypervisor 7.0 Download Center Update3d click on Register. This will then take you to the screen where you need to fill out your details. Once done and confirmed, you will be brought back to the VMware Sphere Hypervisor 7.0 Download Center Update3d page and now you will have license key (greyed out below) and the download link for the hypervisor. Please copy the license key somewhere on your computer so that you can enter it once you are done with the installation.

Now we can almost start the installation. You will notice that you have downloaded a file that is named something like this: VMware-VMvisor-Installer-7.0U3d-19482537.x86_64.iso. This is an image file that needs to be extracted and put on an installation media - in most cases today, that will be a USB key. To do that you will need a program like Rufus or something similar. Now, grab a USB key and insert it in the computer where you have Rufus. Start the app and it should automatically choose your USB key and show it under the Device. Click on SELECT and browse to the location where you downloaded ESXi image from the previous step. It should all look something like this in the end:

Press START and wait for Rufus to create a bootable USB key for you. Once that is done, close Rufus and insert it into your new server. Start the server and, if needed, go to BIOS and change the boot order so that it starts from the USB key. Once the installation starts, first screen you will come to is this:

Press Enter to continue the installation process. Next screen will ask you to accept the EULA, so press F11 to do just that:

On the next screen you will need to choose the installation target. You will be presented with a list of HDDs and/or SSDs in your system. ESXi needs to be installed on one of them to be able to start the server, so you will need to decide where to install it. Be careful where you install it though, as any drive that you choose for installation will no longer be available as a datastore for your VMs.

You have a few options in this case. You can either install it on the smallest drive in your system that is/will be dedicated just for ESXi. Another option is to install it on a different USB key and the last option is to install it on the key from where you are installing it. This will effectively erase the USB key and install ESXi on it. This is possible because ESXi always writes itself into RAM, but this is not an option that I like to use for some reason.

I prefer to use a separate USB key or even the first option (a small drive in the computer) over the USB key overwrite. It may be that I'm wasteful, I don't know. Whatever you choose, the destination drive needs to be at least 8GB (I think) in size.

Select the keyboard layout you're using in the next screen.

Choose and type the root password in the next screen.

You may be greeted with the following warning message in case you have a CPU or some other hardware part that may be outdated in the next version of ESXi (just press Enter to continue):

Finally, confirm that you want to start the installation by pressing F11.

Once the installation is complete, remove the installation USB key from the server, but leave the other one if you decided to install ESXi to a different USB key. Press Enter to reboot the computer.

If everything went well, you should be greeted with a screen like this:

Note a couple of things here. I am running this ESXi in a virtual environment, so basically I'm running ESXi on top of an existing ESXi environment, so that's why the second line says VMware, Inc. VMware7,1.
Since there was no option to set the IP address during the installation process, it picked up an IP address from a DHCP server - in case you don't have one or you just want to change the DHCP-assigned IP address, we will set it up now. Press F2 and enter your root credentials you chose during the installation process.

Under System Customization you have a few options regarding the network setup:

• Configure Management Network
• Restart Management Network
• Test Management Network
• Network Restore Options

Select Configure Management Network and press Enter.

Under Configure Management Network you have a few options - you can list your network interfaces, you can set VLAN and you can change IP settings (both IPv4 and IPv6).
Remember at the beginning of this article when I was explaining different connection scenarios you may have? Well, VLAN (Optional) is what you will be looking at in case you only have one network interface on your server and you still want to use trunking. So in case you configured your switch port something like this:

interface GigabitEthernetX/0/X
 description WhiteBox ESXi trunk
 switchport trunk native vlan 2
 switchport mode trunk
end

Now under VLAN (Optional) you would set the management VLAN to 2. Once this is done, you should then be able to access the management interface via web browser. This is actully incorrect, as native VLAN is not tagged, so there is no need to set any VLAN if you're using native VLAN (July 2023).

One thing I like to do - since I'm not using IPv6 in my network - is to disable IPv6 stack completely. It's up to you whether you want to do that or not - if you want to do it, select IPv6 configuration and choose the first option Disable IPv6 configuration for management network.
Next, choose IPv4 Configuration to reconfigure the DHCP-assigned IPv4 address.
Select Set static IPv4 address and network configuration: by pressing space on your keyboard and then change the IP address to whatever you wish. Press Enter when done to apply these changes.

Now select DNS Configuration to change the hostname of your new server. This is optional and if you want to keep it as localhost, that's up to you :). Please note that you may want to use FQDN if you're going to set up DNS later on (in my scenario, it would be esxi-ntg.demo.networktechguy.com instead of just esxi-ntg (I changed that later on) and finally, press Enter to apply changes.

Next, select Custom DNS Suffixes and enter your own domain that you want to use as a DNS suffix for DNS queries. Note that this doesn't set the domain name for your hostname, it just allows ESXi to use whatever you set here as a suffix for hostnames. Example: instead of ping computer1.demo.networktechguy.com you could enter ping computer1 and ESXi would automatically set it to computer1.demo.networktechguy.com because of this option here.

Finally, press Esc to exit and then Y to apply the changes you made.

If everything was set up correctly, you should be able to access the GUI management interface of your new server by typing the IP address of your server in your browser, something like https://IP_address/.
Since at this point you don't have a signed SSL certificate for your ESXi install, you will have to confirm that you want to access this site by clicking on Advanced and then Accept the Risk and Continue (I'm using Firefox here and it may be a little bit different for you if you're using a different browser).

Finally, you will be greeted with the login screen of your new server. Enter your root credentials and press Log in button.

Once you are logged in, you will be greeted with the Help us improve the VMware Host Client message. If you want to help VMware, go ahead and join the VMware Customer Exoerience Improvement Program by pressing OK, otherwise deselect it and press OK.

Let's go and install the license first. If you don't plan on running this more than 60 days, you can skip this step. Every ESXi system initially runs on a trial Enterprise license.
To apply the license you registered at the beginning, choose Manage on the left, then select Licensing tab and click on Assign License.

Once you paste the license key here, press on Check license and it should come up to this:

If you got the green checkmark, it means that the license is OK and you can assign it to the server at this point. Finally, you click on Assign license and it should now show something like this:

There are two more things that we will do here. First, we will create a datastore that we will use in the future to install all of our VMs. Second, we will create a new virtual switch so that our VMs can talk to the outside world. Please note that creating a new virtual switch is not actually necessary, but I like to separate my physical interfaces into separate virtual switches, so that's what we will do here.

Also, another case when you wouldn't need another virtual switch is if you only have one network interface on your server. Let's start by creating our new datastore. Select Storage from the menu on the left and then on the Datastore tab click on New datastore. Choose Create new VMFS datastore and press Next.

Give the name to the new datastore and select the hard drives that you want to use under this datastore. Click on Next.

Under Select partitioning options make sure that Use full disk and VMFS 6 are selected and press Next.

Review all the settings and confirm by clicking on Finish.

Finally, we will set up the virtual switch and port groups that we will use in our VMs. Think of port groups as VLAN assignments that we would use when configuring physical switch ports.
To do that, select Networking from the menu on the left and select Virtual switches tab. There, click on Add standard virtual swith and give it a name and under security set all options to Accept. Uplink needs to be set to whatever your second, non-management interface is.

Next, we will create port groups. To do that, click on Port groups tab and click on Add port group. First, we will create a trunk port group. A trunk port group is a special port group type that allows us to pass the tags all the way to the VM instead of untagging it and passing untagged frames to the VM. Why do we need this?

Remember when I was talking about virtualizing the firewall earlier? Well, this is when you would need this. Instead of creating a firewall with 1-2-3-4-5-or-whatever number of interfaces, you can just pass all the tags directly to the firewall and then create firewall subinterfaces and manage tags on the firewall itself - that would mean that you would only need a single interface for your firewall!
This is also helpful when you have VLANs that are in your physical infrastructure, but are not configured as port groups in ESXi.

To create a trunk port group, assign VLAN ID 4095 to the new port group:

Make sure that you select the previously create virtual switch under Virtual switch selection, otherwise it will be mapped to the default virtual switch. Finally, repeat the process for all other VLANs that you want to use in your virtual environment. In the end, your list should look something like this:

And that's it! Now you have a working ESXi server and you can start creating VMs. We will cover creating your first VM in the next post.

You can also take a look at this video that shows the whole installation process: