WSL & Docker

Running Ubuntu with WSL ..

WSL & Docker

Using WSL and Docker for LLMs

Windows Subsystem for Linux (WSL) paired with Docker creates a powerful environment for working with Large Language Models. WSL provides a Linux-compatible kernel interface on Windows, allowing developers to run Linux tools directly alongside their Windows applications. When combined with Docker's containerization capabilities, this setup enables consistent deployment of LLM environments across different machines while isolating dependencies.

Docker containers package everything needed to run an LLM—from specific Python versions to specialized CUDA drivers for GPU acceleration—making it straightforward to deploy models like Llama, Mistral, or other open-source LLMs locally. This approach simplifies environment management, reduces "works on my machine" problems, and makes scaling from development to production more seamless, all while preserving Windows as your primary operating system.

Setup WSL

WSL

Windows Subsystem for Linux (WSL) enables developers to run a GNU/Linux environment on Windows. The Ubuntu distribution for WSL is tightly integrated with the Windows OS, supporting features including remote development with popular IDEs and cross-OS file management.

Ubuntu on WSL provides a productive terminal environment that can also be used to launch Linux-native graphical applications.

There are multiple ways of installing distros on WSL:

• Microsoft Store application

• WSL commands run in the terminal

---

PowerShell

1. List available versions.

wsl --list --online

List of Distros

1. Install a version using a NAME from the output

wsl --install -d Ubuntu-24.04

1. You’ll see an indicator of the installation progress in the terminal.

Ubuntu installation

1. Once completed you have access to the WSL subsystem.

WSL

1. Launch Unbuntu 24.04 LTS.

wsl.exe -d Ubuntu-24.04

1. Enter username & password.

Once it has finished its initial setup, you will be prompted to create a username and password.

They don’t need to match your Windows user credentials.

Launch Ubuntu 24.04 LTS

1. Close and open a new Terminal.

2. Check all your currently installed distros and the version of WSL.

wsl -l -v

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\Users\jpore> wsl -l -v
  NAME            STATE           VERSION
* Ubuntu-24.04    Stopped         2
PS C:\Users\jpore>

---

Run and configure Ubuntu

1. Select Ubuntu 24.04 from the drop-down terminal.

Select Ubuntu 24.02 LTS

2. Install the latest updates.

sudo apt update && sudo apt full-upgrade -y

Update & Upgrade

---

Network

WSL doesn't connect to the internet from within the distro. The reason is because Windows automatically generates resolv.conf file with the wrong nameserver.

You may also need to stop WSL from resetting this file when opening future terminals

1. Check if you already have internet connectivity.

ping google.com

1. Check /etc/resolv.conf.

cat /etc/resolv.conf

1. Create a proper resolv.conf.

sudo bash -c 'echo "nameserver 8.8.8.8" > /etc/resolv.conf'
sudo bash -c 'echo "nameserver 8.8.4.4" >> /etc/resolv.conf'

1. Generate a new /etc/wsl.conf file. This will prevent the resolv.conf being recreated on each session.

sudo bash -c 'cat > /etc/wsl.conf << EOF
[network]
generateResolvConf = false
EOF'

1. Restart WSL.

wsl --shutdown

Ubuntu Desktop

Ubuntu Desktops

Here's a brief overview of each desktop environment:

GNOME: Modern, streamlined desktop environment with a minimalist approach and strong touch-screen support. It prioritizes simplicity over customization, using a unique workflow design.

Xfce: Lightweight and traditional desktop environment focusing on speed and low system resource usage. Known for its reliability and conventional desktop layout with a Windows-like feel.

GNOME Flashback: A classic version of GNOME that mimics the traditional GNOME 2 interface. Provides a familiar desktop experience with panels and menus while using fewer system resources than standard GNOME.

KDE Plasma: Highly customizable desktop environment with a modern, polished look and extensive features. Offers widgets, effects, and themes while maintaining good performance.

Cinnamon: Created by Linux Mint team, offering a traditional desktop layout with modern features. Combines conventional usability with contemporary graphics and effects.

MATE: Fork of GNOME 2, providing a classic desktop experience with modern technology support. Ideal for users who prefer traditional desktop metaphors and workflows.

LXDE: Extremely lightweight desktop environment designed for low-resource computers. Focuses on speed and energy efficiency while maintaining basic functionality.

LXQt: Modern Qt-based continuation of LXDE, offering a lightweight experience with a more contemporary look. Provides better visual integration with Qt applications while remaining resource-efficient.

1. Update & upgrade.

sudo apt update && sudo apt upgrade

1. Install Tasksel - installs Desktops

sudo apt install tasksel

1. Run tasksel.

sudo tasksel

1. Select a Desktop. Up & down arrow keys + Spacebar - Xfce is lightweight

Select desktop

wait a few minutes ..

1. Install remote desktop server.

sudo apt install xrdp

1. Check the IP address of the VM.

ip a

IP of VM

1. RDP into the VM.

RDP

1. Save connection details & Connect. You will be asked again for Password.

Xrdp

You can enter commands either via:

• host Visual Studio Code

• Ubuntu 24.0 terminal

• RDP into the Xfce

1. Download and arrange your apps in the Taskbar - Terminal & Chrome.

Docker

Docker

Installing Docker on Ubuntu 24.0.4 extends its capabilities by enabling users to harness the power of containerization for managing and deploying applications more efficiently.

1. Select Ubuntu 24.04 from the drop-down terminal.

2. Update Package Index.

sudo apt update && sudo apt upgrade

1. Install Docker Engine.

sudo apt install docker.io

1. Adding Current User to Docker Group.

sudo usermod -aG docker ${USER}

1. Log out and back in.

2. Check User belongs to Docker group.

id ${USER}

1. Test the Installation.

docker run hello-world

'Hello World'

---

Troubleshooting

If you get the following error message:

chatbot@Office:~$ docker run hello-world
docker: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.

Run the following command:

newgrp docker

---

Docker Compose

1. Ensure your package list is up to date.

sudo apt update -y && sudo apt upgrade -y

1. Install Docker Compose.

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

1. Check Docker Compose installation.

docker-compose --version

Docker Compose version v2.33.1

---

Return to this section once you have installed the CUDA toolkit.

This section configures the docker daemon to use GPUs.

1. Configure Docker daemon to use Nvidia runtime.

# Configure the Docker daemon to use NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker

# Restart Docker to apply changes
sudo systemctl restart docker

1. Verify your NVIDIA driver installation.

docker run --rm --gpus all nvidia/cuda:12.0.1-base-ubuntu22.04 nvidia-smi

Check GPUs

---

Troubleshooting

if you get the following error message.

The error message indicates that the nvidia-ctk command is not found on your system. This tool is part of the NVIDIA Container Toolkit, which is needed to enable GPU support in Docker containers.

1. First, set up the repository.

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

1. Update the package list.

sudo apt update

1. Install the NVIDIA Container Toolkit.

sudo apt install -y nvidia-container-toolkit

1. After installation, you can configure Docker to use the NVIDIA runtime.

sudo nvidia-ctk runtime configure --runtime=docker

1. Restart Docker to apply the changes.

sudo systemctl restart docker

If you're using WSL2, you might need to slightly modify these steps since systemd might not be fully enabled. In that case, you might need to use.

sudo service docker restart

CUDA Toolkit

CUDA Toolkit

CUDA (Compute Unified Device Architecture) is a parallel computing platform developed by NVIDIA. It provides an application programming interface (API) that enables developers to harness the immense processing power of NVIDIA graphics processing units (GPUs) for general-purpose computing.

CUDA cores are specialized processing units within the GPU optimized for parallel computing tasks. They are capable of executing multiple threads concurrently.

1. Install ubuntu-common-drivers which will help identify the required NVidia driver.

sudo apt install ubuntu-drivers-common

ubuntu-common

1. Detect devices.

sudo ubuntu-drivers devices

// Some code

1. Install the suggested driver - Ubuntu 24.04.

sudo apt install nvidia-utils-550

1. Reboot the system.

sudo reboot now

If no driver is suggested run the following command.

In this case, the NVIDIA driver was already installed by default on Ubuntu.

1. Verify installation by accessing Nvidia System Management Interface.

watch nvidia-smi

Check GPUs

CUDA Toolkit is not yet installed ..

1. Install the gcc compiler.

sudo apt install gcc

Install gcc

To install the toolkit follow the instructions outlined in the link below.

Select

1. Introduction — Installation Guide for Linux 12.8 documentation

Select platform

1. The following installation instructions are copied over .. and run.

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.8.1/local_installers/cuda-repo-wsl-ubuntu-12-8-local_12.8.1-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-12-8-local_12.8.1-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-12-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-8

Installation of CUDA Toolkit

1. Reboot the system to load CUDA.

sudo reboot now

1. Update the environment variables.

nano ~/.bashrc

1. Copy and paste the following to the bottom of the file.

# CUDA Toolkit
export PATH=$PATH:/usr/local/cuda-12.2/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.2/lib64

1. Save.

CTRL + o
Enter
CTRL + x

1. Reload the file.

. ~/.bashrc

1. Test CUDA.

nvcc -V

Nvidia Complier

Remember to finish configuring the docker daemon to use the Nvidia runtime.

All set ..

The compiler is essential for developers who want to leverage NVIDIA GPUs for parallel computing tasks, as it handles the complex process of preparing code to run efficiently on both CPU and GPU hardware.

---

Troubleshooting

You may have some issues download some packages using IPv6.

1. Create the IPv4 force configuration file:

echo 'Acquire::ForceIPv4 "true";' | sudo tee /etc/apt/apt.conf.d/99force-ipv4

1. Clean the apt cache and update:

sudo apt clean && sudo apt update

1. Now try installing alsa-utils again:

sudo apt install alsa-utils

After this completes successfully, we can proceed with the CUDA driver installation. You'll be able to run sudo ubuntu-drivers devices without the aplay error.

Cursor

Cursor

Take your vibe coding to the next level ..

Cursor.ai is an AI-powered code editor built for modern developers to enhance coding efficiency. It seamlessly integrates with popular language models to provide intelligent code completion, generation, and editing capabilities.

Cursor offers features like chat-based code assistance, automatic documentation generation, and contextual code understanding to help developers write, understand, and debug code more effectively.

1. Download Cursor for your host environment.

Downloads | Cursor - The AI Code Editor

1. Once installed, connect to the WSL -Ubuntu 24.0.4

Connect to WSL

1. Clone the Workshop--LLM directory.

gh repo clone jporeilly/Workshop--LLM

1. In Cursor, connect to WSL and Open folder: Workshop--LLM.

Cursor UI

From here you can switch between using Cursor to sync any changes on host and WSL.