Overview written by Martin Ågren - Updated: October 6, 2026
"Open" = Digital technology that is publicly accessible for inspection and use is "open".
"Closed" / Proprietary = Digital technology that is not publicly accessible for inspection and use is "closed".
Weights = Strength of connections between NN nodes, learned through ML. - Crucial for an LLM’s output quality.
Open weights = An AI model’s weights that are “open,” meaning those weights are publicly available for inspection and use.
Open-weight model = AI model whose weights are publicly available. - The model can be downloaded, used & often modified.
Closed model = AI model whose trained weights are not publicly available for inspection or use, e.g. LLMs in ChatGPT.
Open-source = Historically open source code. - Nowadays: Primarily license related. - Ex: "This model is under the MIT License".
Open-weight + Open-source = Open-weight models are often released under licenses, many of which are open-source licenses.
Hardware demands = Running performant open-weight model requires powerful hardware: GPU, RAM, CPU, SSD, PSU, cooling.
GPU = Graphics Processing Unit - Processor specialized for parallel calculations. - Crucial for running/training AI models fast.
NVIDIA = Leading designer of GPUs used for AI training and inference. - Jensen Huang: CEO, "president", co-founder.
VRAM = Video Random Access Memory - The more VRAM, the better - Load model into this GPU memory to run AI model fast.
Download model = An open weights model needs to be downloaded from the internet. - e.g. from HuggingFace.co / within app
Swedish = An LLM for your computer that is good at generating output in Swedish? - Perhaps try: Apertus-8B-Europa-SFT-mix.
Inference = When a trained AI model produces output from input. - Ex: ChatGPT generates answer.
Local inference = Run open-weights model on device (e.g. computer) or on own server - inference on your local/own hardware
Statistics, downloads from Hugging Face.
Blue field represents downloaded language models.
Screenshot taken: October 6, 2026
Source: https://huggingface.co/spaces/modelpulse/model-pulse
Local inference, on own device:
- E.g. on own computer
- No input data needs to leave the device.
- No internet connection is needed for running the AI model.
Local inference, on own server:
- Own server, "on-prem", on-premises
- No input data needs to leave the organization.
Cloud inference, self-managed inference, open-weight model:
- Concept: Cloud infrastructure with GPU compute (dedicated graphics card - or shared GPU)
- Examples: Dedicated GPU server / Cloud GPU Virtual Machine / GPU container (e.g. runs in a Kubernetes pod)
- Cloud providers = Azure (Microsoft) / AWS (Amazon) / Berget (Swedish provider, servers in Sweden)
- You upload the model yourself & install runtime for inference etc.
- GDPR = MS & Amazon offers DPA agreement - Risk: CLOUD Act: Provider can be forced to hand over data (even if stored in EU)
Cloud inference, managed inference, open-weight model:
- Server in Sweden: "Berget" - You use API & Berget runs the model.
- Server in US or EU: Hugging Face / OpenRouter
- Groq (for speed) - US/EU, but EU-only data processing not guaranteed
( Closed frontier cloud models = ChatGPT / Claude / etc. )
- I used brackets to mark that this is NOT running open-weight model.
Data security = Input data stays local, within org. - Risk: Connecting model to external system: Data could be sent outside org...
- Example 1: I heard a Swedish developer who were instructed to only use local inference on computer (maximum data security)
- Ex 2: Riksbanken (central bank) will use local inf for sensitive info like identify risks in the financial system (Nyhetsbyrån direkt)
GDPR = Personal data does not leave the org - improves GDPR compliance by keeping personal data within the organization.
Sovereignty = Goal: Control over own digital tech - Ex: No over-reliance on US tech - Many in EU are worried
Availability = Avoid slow external servers + Offline use possible
Speed = Powerful hardware for great speed - Ex: Faster than ChatGPT
Cost optimization = No subscription + No external API calls - Ex: Hybrid approach: Local (simple task) & Frontier m (hard task)
Customization = Greater technical control - Ex: RAG / Supervised fine-tuning (tip: Try system prompt: Good enough result?)
Environment?... = Could be better for the environment under the right circumstances.
Ollama = For LLM on computer - Application providing runtime for local inference.
Ollama.com > Download EXE (if Win) > Run EXE > Bg process active even after Win restart > P.Shell: Get model, Run AI, Chat
LM Studio CLASSIC = For LLMs - Company: Element Labs - LMstudio.ai > Choose "LM Studio" (i.e. classic version).
LM Studio BIONIC = A newer version of LM Studio - For agentic local inference: Define a goal, AI make decisions to reach goal.
Stability Matrix = For image generation - Company: Lykos AI - For PC & Mac - Free & Open-source - Download model inside app
LTX Desktop = For video generation - Company: Lightricks - For PC & Mac - Free & Open-source - Download model inside app
ACE-Step Studio = For music gen - Companies: ACE Studio & StepFun - PC & Mac - Free & Open-source - Download model in app
LLMs to download = Qwen (Alibaba) / Kimi (Moonshot AI) / GLM (Z.ai) / GPT-OSS (OpenAI) / Mistral / Llama (Meta) / DeepSeek
LLM for Swedish? = LLM for computer that is good at generating Swedish? - Perhaps try: Apertus-8B-Europa-SFT-mix.
Image generation model = Flux-dev (Black Forest Labs) for high quality - Z-Image-Turbo (Alibaba) for high speed
Video generation model = LTX Fast (Company: Lightricks)
Music generation model = ACE-Step (Companies: ACE Studio & StepFun) - HappyShrimp (Alibaba)
B = Number of billion parameters, e.g. "30B" - More parameters => improved capability, stronger hardware needed
Size (most common interpretation) = Number of parameters, e.g. "30B"
Size (alternative interpretation): Storage size, e.g. 19 GB
GGUF = Popular file format for AI models
Hybrid model = Supports multiple modes: Reasoning / Non-reasoning ("Standard")
Instruct = Fine-tuned model, prepared for conversations
MoE = Mixture of Experts - Only parts of AI model is active/activated/used during inference - Fast model, but a bit less capable
Dense model = In contrast to a MoE model, all parameters are used during inference.
Q4 / Q5 / Q6 / Q8 = Level of quantization - Lower number: more quantized, faster, smaller file size, less capable.
- Bits: Q4 means that the number of bits per weight is 4.
- Starting point: Q4 is often a good starting point for trying a new open-weight LLM: if Q4 is fast, then try Q6 etc.
- Diminishing returns: Quality difference is higher between Q4-Q5 compared to Q5-Q6.
TTFT = Time To First Token = Example: For my desktop computer: About 0.3 seconds which feels really fast.
Speed / Output speed / Generation speed = Measured in (tok/s (tokens per second). Example: About 55 tok/s for me (feels fast).
Computer / Server ? = Have you tried local inference and if so: Did you try it on computer or on server?
Model? = What model do you like best for local inference?
App? = What application did you use? LM Studio? Ollama?