Setting Up Local LLMs on USB

First, you'll need a USB 3.0 flash drive, I recommend 64GB or larger, so you can store more LLMs on it. Format the drive using the exFAT file system. Right-click the drive,

choose Format, and select exFAT as the file system. For faster loading, plug the flash drive into a USB 3.0 port. Next, go to the LLamaFile GitHub page. Click the Releases link on the right.

The newest release will show at the top. Scroll down to the Assets section and find the file that starts with LLamaFile, followed by the version number, like LLamaFile 0.93. Click it to download.

After the file downloads, rename it to LLamaFile.exe and move it to your USB drive. After this is done, go to the Hugging Face website. Click on Models,

then select GGUF in the left navigation bar under Libraries. Here you can search for models and sort them by categories. Pick a model you like, but keep it 14 billion parameters, or smaller if it's a dense model,

or 30 billion or smaller for MOE models. I'll search for the QIN38B model. Once you find the model, click on it, go to the Files and Versions tab at the top,

and download your version. I will download QIN38B Quantized Q4 version. After the download is done, move the GGUF file to your USB flash drive. And the final step before running a local LLM

from a USB flash drive is to create a simple .bat file to start it. Open Notepad or any text editor, type the following command, or copy and paste it from the description.

Then, enter the full model file name exactly as it appears on your USB flash drive, including the .GGUF extension. Save this file to the USB flash drive with a .bat extension, and name it something that helps you identify the model,

like run QIN38B.bat, then run the file. The model will load into system memory. This can take a minute or more depending on the model size and your computer. On my desktop with 128 GB of DDR4 memory

and Ryzen 7 5800X processor, it took 34 seconds to launch. Once the load is done, LLM file will open in your browser. For this tutorial,

I'll adjust only the system prompt, user name, and bot name. I'll leave all other settings at their default values and start chatting by entering a question

in the box at the bottom. The QIN3-8B-Q4 model responded at 7.15 TPS. Let's try another model. This time, I'll use QIN3-30B-A3B-Q4. It's a MOE model with 30 billion parameters.

This model responded at 15.67 TPS. That's it, a fully portable setup for running local LLMs from a USB flash drive. No installation, no internet, and no admin access required. Just plug it into any Windows PC and start using it.

This setup is private, beginner-friendly, and works entirely offline. You can switch between different models and different computers and run your own AI assistant directly from the USB drive. Feel free to share your experience or tips in the comments. Thank you for watching.