OpenAI has broadened the capabilities of GPT-4o by introducing vision fine-tuning, enabling developers to refine the model using images in addition to text. This enhancement allows for improved image understanding, which could benefit various applications such as enhanced visual search, autonomous vehicles, and medical image analysis.
Previously, developers could only customize models with text datasets, leading to limited performance improvements for tasks requiring image comprehension. The new vision fine-tuning process is straightforward: developers format their image datasets accordingly and upload them to the platform. Improvements can be seen with as few as 100 images, with larger datasets potentially offering even greater performance enhancements.
OpenAI has collaborated with partners to explore practical applications of vision fine-tuning. Grab, a prominent rideshare company in Southeast Asia, has successfully implemented the new feature to refine its mapping data. By fine-tuning GPT-4o with only 100 examples, Grab enhanced lane count accuracy by 20% and improved speed limit sign localization by 13%.
Similarly, Automat, an enterprise automation firm, used vision fine-tuning to increase the success rate of its desktop bots in locating UI elements based on natural language descriptions. The result was a significant improvement from a 16.60% success rate to 61.67%, showcasing a 272% increase in performance.
Coframe, an AI-focused company, leverages vision fine-tuning to enhance digital content creation. By tasking GPT-4o to generate website sections based on image and code inputs, Coframe improved stylistic consistency and layout correctness by 26%.
Ensuring safety and privacy remains a top priority. OpenAI commits to running automated safety evaluations and ensures that fine-tuned models comply with usage policies. Developers retain full control over their business data, and OpenAI’s models do not incorporate this data without explicit permission.
Vision fine-tuning is available for developers on paid usage tiers, with a 1 million-token training allowance per day available for free until October 31, 2024. Afterwards, training will cost $25 per million tokens, with different rates for input and output tokens. To begin, developers can visit the fine-tuning dashboard and select the appropriate base model.
For more detailed information, you can read the original article here: https://openai.com/index/introducing-vision-to-the-fine-tuning-api/
