Local LLMs: deployment and operation in corporate infrastructure (AILOC)

Course content:

• Local model fundamentals
1. What is a local LLM and how inference works
2. Local model versus cloud API and on-premise service
3. Open source, open-weight and proprietary models
4. Model parameters, layers, tokens and context window
5. VRAM, RAM, CPU, GPU and storage requirements
6. Quantization: FP16, BF16, INT8, INT4 and GGUF
7. Impact of model size on quality and performance
8. Chat, base, reasoning, coding and multimodal models
9. Embedding and reranking models
10. Inference, RAG, fine-tuning, LoRA and QLoRA
11. Model selection by task, language, licence and hardware
12. Benchmarks, evaluation and practical testing
13. Limitations of local models and common errors
14. Security and data protection

• Hardware and operating environment
1. Workstation versus corporate server
2. NVIDIA, AMD, Intel and CUDA
3. VRAM, multiple GPUs, CPU and hybrid inference
4. Linux, Windows, WSL, Docker and Kubernetes
5. Performance, power consumption, cooling and monitoring
6. Infrastructure benchmarking and sizing
7. Unified Memory platforms, Strix Halo platform, Mac Studio, RTX Spark, DGX Spark and more

• Ollama
1. Rapid model deployment on a workstation
2. Model management and basic configuration
3. API for connecting applications

• llama.cpp
1. GGUF and quantized models
2. CPU, GPU and various backends
3. llama-cli and llama-server
4. Offloading layers to the GPU
5. Configuring context and inference parameters
6. OpenAI-compatible API
7. Operation on workstations and servers
8. Basic optimization and benchmarking

• vLLM
1. vLLM architecture
2. OpenAI-compatible API
3. PagedAttention and continuous batching
4. High throughput for multiple users
5. Tensor parallelism and multiple GPUs
6. Prefix caching and speculative decoding
7. Quantized and multimodal models
8. LoRA adapters
9. Deployment in Docker and Kubernetes
10. Monitoring and production operation

• GPUStack
1. Managing GPU workers and models
2. Deployment across multiple servers and GPUs
3. Resource allocation, monitoring and scaling
4. Integration with inference engines

• Open WebUI
1. Connecting Ollama, llama.cpp and vLLM
2. Managing users and models
3. Knowledge base and RAG
4. Tool integration and internal chatbot

• Open source models
1. Qwen, Gemma, Llama, Mistral, DeepSeek, GPT-OSS and more
2. Coding, reasoning and multimodal models
3. Current developments in open source models
4. Licensing and commercial use
5. Benchmarks and practical model selection
6. Quantization, LoRA and fine-tuning

• Integration with development and other tools
1. VS Code
2. Aider
3. IntelliJ IDEA
4. Open Code
5. OpenClaw
6. Continue, Cline, Roo Code and more
7. Local coding assistant
8. Agent workflows and tool integration

• Corporate operation and security
1. Internal APIs and model sharing
2. Authentication and SSO
3. Protection of sensitive data
4. Prompt injection and agent security
5. Monitoring, backups and updates
6. RAG over corporate documents
7. Hybrid local and cloud architecture