Vision Language Model (VLM)

Vision Language Models combine computer vision with language understanding. You can show the model an image and ask questions about it in natural language. GPT-5.2, Claude Opus 4.6, and Gemini 3.0 all have VLM capabilities. This enables entirely new application categories. How they work: images are converted into visual tokens using a vision encoder (typically a Vision Transformer / ViT). These visual tokens are then processed alongside text tokens by the language model. The model learns to associate visual features with language concepts during training on image-text pairs. Practical applications are exploding: (1) Document understanding — extract data from invoices, receipts, forms, including handwriting. (2) UI analysis — describe what is on a screen, enabling AI-powered testing and accessibility. (3) Medical imaging — analyze X-rays, dermatology photos, pathology slides (with appropriate disclaimers). (4) Retail — identify products from photos, compare items visually. (5) Content moderation — detect inappropriate images with nuanced understanding. For builders: VLMs are accessed the same way as text-only models, just include images in your API request. Costs are higher (images use more tokens — a typical photo is 1,000-2,000 tokens). Optimize by resizing images before sending (most VLMs work well with 1024x1024 or smaller) and using the "low detail" mode when full resolution is not needed.