Jan.ai

One stop shop to get up and running ..

Introduction

In this workshop we're gong to implement a simple solution using existing open-source applications.

• install Jan.ai on Windows

• choose the right model for your hardware

• difference between Q4, Q8, and GGUF models

• run Qwen, DeepSeek AI, and other AI models locally

This section is for those that have the resources to run local LLMs.

Jan.ai

Jan.ai

Jan.ai

Most people think running AI models locally is complicated. It's not. Anyone can run powerful AI models like DeepSeek, Llama, and Mistral on their own computer.

This guide will show you how, even if you've never written a line of code.

Jan uses llama.cpp, an inference that makes AI models run efficiently on regular computers. It's like a translator that helps AI models speak your computer's language, making them run faster and use less memory.

1. Download Jan.ai.

Jan | Open-source ChatGPT Alternativejanframework

Link to Jan.ai

Jan.ai

Windows

1. Double-click on the .exe

2. Choose a model that fits your hardware: Click on the Hub option in the left menu bar.

Take a look at the different models ..

For example for coding - qwen2.5 7B coder Instruct Q4

Models

Jan will indicate if a model might be Slow on your device or Not enough GPU / RAM based on your system specifications.

Machine	Model Size	Description
Laptop 16GB	3B - 7B models	Like having a helpful assistant
Laptop 32GB	7B - 13B models	Better at complex tasks like coding and analysis
Desktop 64GB	13B+ models	Great for professional work and advanced tasks

These are different versions of the same AI model, just packaged differently to work better on different computers:

• Q4 versions: Like a "lite" version of an app - runs fast and works on most computers

• Q6 versions: The "standard" version - good balance of speed and quality

• Q8 versions: The "premium" version - highest quality but needs a more powerful computer

1. Once downloaded you're good to go ..

Jan.ai - System Monitor

1. If you're lucky enough to have some Graphics cards then in Settings > Advanced Settings you can enable GPU Acceleration.

Advanced settings

Features

Features

Assistant

Assistant is a configuration profile that determines how the AI should behave and respond to your inputs. It consists of:

• A set of instructions that guide the AI's behaviour

• Model settings for AI responses

• Tool configurations (like knowledge retrieval settings)

By modifying assistant instructions, you can customize how Jan understands and responds to your queries, what context it should consider, and how it should format its responses.

1. In any Thread, click the Assistant tab in the right sidebar

2. Enter your custom instructions in Instructions input field.

Use the model to define your Assistant.

Act as a software development mentor focused on Python and JavaScript. 
Provide detailed explanations with code examples when relevant.
Use markdown formatting for code blocks.

Assistant

1. Your instructions will be applied to the current thread right after you click out of the instruction field .

Model

You can customize the following parameter types:

Inference Parameters: Control how the model generates responses

Model Parameters: Define the model's core properties and capabilities

Engine Parameters: Configure how the model runs on your hardware

x

x

Inference

x

These settings determine how the model generates and formats its outputs.

Parameter	Description
Temperature	- Controls response randomness. - Lower values (0.0-0.5) give focused, deterministic outputs. Higher values (0.8-2.0) produce more creative, varied responses.
Top P	- Sets the cumulative probability threshold for token selection. - Lower values (0.1-0.7) make responses more focused and conservative. Higher values (0.8-1.0) allow more diverse word choices.
Stream	- Enables real-time response streaming.
Max Tokens	- Limits the length of the model's response. - A higher limit benefits detailed and complex responses, while a lower limit helps maintain conciseness.
Stop Sequences	- Defines tokens or phrases that will end the model's response. - Use common concluding phrases or tokens specific to your application’s domain to ensure outputs terminate appropriately.
Frequency Penalty	- Reduces word repetition. - Higher values (0.5-2.0) encourage more varied language. Useful for creative writing and content generation.
Presence Penalty	- Encourages the model to explore new topics. - Higher values (0.5-2.0) help prevent the model from fixating on already-discussed subjects.

x

x

x

Model

x

Engine

Tools

Retrieval

Chat with your documents and images using Jan's RAG (Retrieval-Augmented Generation) capability.

This feature is currently experimental and must be enabled through Experimental Mode in Advanced Settings.

1. In any Thread, click the Tools tab in right sidebar.

2. Enable Retrieval.

1. Once enabled, you should be able to upload file(s) & image(s) from threads input field

Experimental feature: Can only handle PDFs that are small and simply structured.