Niantic has taken a significant leap in augmented reality (AR) by integrating Meta Llama 2 into its latest AR game, Peridot. Known for its popular AR games like Pokémon GO, Niantic’s Peridot enables players to interact with lifelike digital pets, known as “Dots,” that exhibit smart and unpredictable behaviors. Llama 2 powers these interactions, making each Dot’s responses dynamic and unique in real-time.
Asim Ahmed, Global Marketing Lead for Peridot at Niantic, stressed the importance of leveraging large language models (LLMs) like Llama to create immersive and realistic gameplay. Rather than relying on manually programmed, limited reactions, Niantic uses Llama 2 to generate appropriate responses from a vast animation library, effectively simulating the spontaneous nature of actual pets.
Niantic’s decision to use an open-source model like Llama 2 accelerated development, enabling the Peridot team to create more profound player connections swiftly. The flexibility of Llama’s framework facilitated the quick prototyping and iteration needed to handle dynamic interactions between creatures and their environment while maintaining data privacy.
Introduced in November 2023, Llama 2 allowed Dots to react suitably to their surroundings using an advanced AR recognition system and extensive animation library without fine-tuning the model itself. Niantic’s development team faced challenges in crafting prompts expressive enough to capture each Dot’s personality traits while maintaining prompt response times. Through continuous refinements, they enabled Dots to exhibit varied and unpredictable behaviors, enhancing the game’s sense of realism.
Further advancements have since allowed players to have “conversations” with their Dots, creating more personalized and engaging gameplay experiences. Llama’s latest integration supports spontaneous AI-driven responses, adding layers of joy, curiosity, or even mischief to the interactions, making each session both thrilling and lifelike.
Niantic also employs advanced computer vision algorithms to convert images captured by a player’s camera into accurate 3D models, allowing Dots to interact realistically with their surroundings. This interaction is facilitated by the Niantic Lightship ARDK, enabling Dots to recognize and respond to physical objects like flowers, food, and pets. Each Dot’s unique personality traits, such as history and preferences, guide its reactions, enhanced by Llama 2’s processing capabilities.
Despite the complexity, Niantic successfully addressed challenges such as latency in input comprehension by defining response formats in JSON. This approach significantly improved interactions, creating lifelike and unpredictable behaviors from the virtual pets.
Looking ahead, Niantic aims to further evolve Peridot, enhancing player interactions across various devices and continually pushing the boundaries of what is possible with AI and AR. As Ahmed notes, the success of generative AI in Peridot gives a glimpse into the future possibilities, positioning Niantic to keep innovating in the AR space.
You can read the original article here: https://ai.meta.com/blog/niantic-peridot-llama/
