AI agent breaks AI generated Minecraft
AI agent breaks AI generated Minecraft
We heard you liked AI so we put an AI agent in your AI minecraft.
Earlier this week Decart and Etched released Oasis, a generative AI model that learns to generate a Minecraft-like game purely by learning from videos of Minecraft, with no coding.
We are building foundation models for behavior in 3d worlds. Since our first product is an AI agent that plays Minecraft, we thought it would be fun to drop it into this imagined game of Minecraft.
Conventional Minecraft agents, such as Voyager, depend on language and API interfaces, making it challenging to operate in AI-generated versions of Minecraft. In contrast, this was pretty easy for us since some parts of our agent actually understand the world and play the game purely from vision. As a result, our agents can navigate and interact within these AI-generated landscapes. You can see the results compiled into a video.
Unexpectedly, the two AI models (our agent playing, and the Oasis model generating the world) tend to collapse into noise if you let them run for more than a few seconds. This is probably because both models were experiencing out-of-distribution data compared to what they were trained on1. But, before that happens we think this shows a glimpse of a future, where AI’s imagine worlds and in those worlds both humans and AIs will participate and collaborate together.
In 2017 AI generated images looked like this:
Now, of course, AI generated photorealistic images are routine. We think the short clips above are the pixelated equivalents for AI worlds with AI agents.
For those interested in experiencing AI companions in the standard Minecraft environment, our AI agents actually play much better there. Download our app from elefant.gg and explore the future of gaming with AI today.
Oasis only released their small model, likely the larger model they have not released would stay coherent for longer. We are happy to try it if/when it becomes available.