Homelab to Hackathon: GenAI for TAS in a box!

Yes- that’s right, for part of last year’s VMware Explore Barcelona Hackathon project for Team TAS, I decided to take my ~~love~~ obsession 😅 with building homelab gear to the next level.

We Put GenAI for Tanzu Application Service in an 8-liter box!

If you are unfamiliar with Tanzu Application Service, it’s a modern app platform based on Cloud Foundry. At Tanzu, we often build bolt-on features or “tiles” to the platform to extend the functionality of TAS. GenAI for TAS allows platform engineers to serve private local language models (LLMs) rapidly, making it easy for developers to integrate into their applications via the service marketplace.

Why did I build a mobile #homelab?

My main goal was to build a machine for our team to host a small runtime Tanzu Application Service (TAS) on vSphere, our beta GenAI for Tanzu Application Service tile, and the newly launched Tanzu Postgres Tile. I wanted to show how such a powerful solution could run on a single machine! But this machine also had to fit in a suitcase for international travel. I could have easily lugged one of my HP z440s, but getting one of those beasts through TSA and customs sounded like a non-starter.

Now, like all good #homelabbers, I spent waaaay too much time reading blogs from William Lam and the rest of the vCommunity. After relentless searching on eBay, Amazon, and Newegg, I settled on a BOM that fit the bill!

Everything is waiting to be assembled.

The BOM:

Chassis: ASROCK DESK MEET X300
I picked this beast because this is the best option to:

Support for 128 GB of ram
Have something that fits cleanly in a suitcase.

I could have gone with the intel version (ASROCK DESK MEET B66), but dealing with the intel little core / big core situation with ESXi seemed like a pain. Going into this, I knew that the onboard NIC on the AMD version would not work out of the box, but I planned to address it with USB NICs.

CPU: AMD Ryzen 5700G (8c/16t) APU
This was a simple decision; I went with the highest core count processor I could put in the Desk Meet X300 and still have integrated graphics support. I planned on using the PCIe slot with a Telsa card that has no video output, hence the need for integrated graphics.

RAM: 2 * Crucial Pro Ram 64GB Kit DDR 3200MT/s
I needed to max out the box with 4 “32gb” DIMMs of DDR4. I got two of these kits on a Black Friday sale on Amazon.

STORAGE (local VMFS): SK hynix Gold P31 2TB NVMe
Another Black Friday deal – I grabbed this NVMe drive to use as my primary datastore. Since this was a single-node deployment, I just opted for local VMFS. This drive works well, and ESXi has no issues detecting it.

Storage (ESXi Boot): TEAMGROUP T-Force Vulcan 240gb sata SSD
Nothing too crazy here, I really like these Teamgroup SSDs for ESXi boot, they are super cheap and just seem to work.

GPU: Nvidia Telsa T4
I was very limited on what supported card models could physically fit; the Desk Meet only supports 20CM of clearance for a graphics card. It boiled down to either the Nvidia Telsa T4 or the newer version known as the Nvidia A2, as I needed to play in the “enterprise” lineup for Nvidia cards. This part is crucial to running local large language models in BOSH and Tanzu Application Service. I was able to pick up one second-hand on eBay for ~700 USD.

Excuse the cabling mess – but I wanted to show the compact size of the box. See the “I ❤️ TAS” sticker for scale 😎.

I removed the faceplate of the T4 to expose the heatsink directly to this 140 mm fan; I squeezed the fan in by consuming the extra slot plus a little free room left over in the case. The T4 was designed for a server chassis and expects forced air from the server itself instead of providing its own fan.

NIC: Cable Matters 2-Pack PnP USB to Ethernet Adapter

I was able to get these adapters working at full speed with the USB nic fling for ESXi, but they do work at 100M without any additional driver being loaded into ESXi. Big shout out to William Lam’s blog for helping me sort through all the potential options.

Networking: UniFi Dream Router

I needed a simple network stack for my deployment to run Wi-Fi and VLANs. UniFi gear is my preference, and the Dream Router was simple and easy to set up. I didn’t have to worry about networking with this device.

The complete solution is up and running in my hotel room – the night before the hackathon!

Running GenAI for TAS on the mobile lab:

You may be wondering what exactly GenAI for TAS looks like. Or why did I want to bring it to the VMware Explore Hackathon in Barcelona? That answer is relatively simple: Team TAS wanted to build off the success of the VMware Explore Hackathon in Las Vegas — where we took first place with the original POC of Gen AI for TAS. I definitely want to give a big shout to my teammates Long Nguyen, GenAI for TAS’s lead engineer, and Amelia Downs, who traveled to both hackathons, co-presented with me, and dealt with me lugging all this gear around the streets of Barcelona. 😅

Here’s a quick recap of what we ran at the hackathon on the mobile lab:

vCenter – “our support IaaS” of choice for TAS here.
Tanzu Operations Manager – The Ops Man VM bootstraps BOSH and Tanzu Application Service.
pfSense – for local DNS/NTP
Nvidia’s NLS server – Nvidia’s license server to support vGPU support
Small Runtime Tanzu Application Service – The compact version of TAS that can fit in an 8L host!
Tanzu Postgres tile – for on-demand database-as-a-service. Postgres was critical in enabling “embeddings” via the pgvector extension.
GenAI for TAS tile: The tile enables deployment of LLMs and an OpenAI-compatible API so developers can easily wire up applications and build chatbots on a completely private GenAI infrastructure. See the architecture diagram for GenAI for TAS below.

We successfully got everything up and running on my “mobile lab.” From a vSphere perspective, the host was able to run everything with about 5-6 GB of RAM free. 😅

We pushed the modified version of Spring Music with a “Spring Music Assitant” embedded into it using SpringAI. This app lets us make use of the local Large model hosted by the GenAI for TAS tile to private calls to our LLM talk to our Tanzu Postgres database and use that for embeddings to supplement the LLM.

Overall, both hackathons were a great experience for team TAS, and I’m thrilled that we were able to continue this and drive to launch our beta last week!

I plan on doing a deep dive into using GenAI for TAS in another blog post – so please stay tuned for more information; in the meantime, you can check out my demo to see GenAI for TAS in action: