The game this article is based on is “Baba Is You” [1]. It’s a logic puzzle game where you try to get “you” to “flag” — or more precisely, “win” (which, by default most of the time, “flag is win”) — but you manipulate a number of objects and join them with other words to change the “rules” of the game.
Something that set Baba is You apart for me compared to any other puzzle game I can remember playing is the puzzles themselves were actually "funny." Not like Portal, where you solved puzzles alongside a funny narrative, but in the actually language of the puzzles themselves there are setups which get subverted in absurd and delightful ways, often in multiple layers, as you work your way through to a solution. Playing made me feel like the math part of my brain was laughing.
The game doesn't hold your hand, and I think it took about 10 hours for me to learn enough of the puzzle language for things to get really good (and then it started to descend into frustrating fiddliness in the deep endgame), but the middle 30 hours or so we're some of the most gratifying gameplay I've experienced.
It kind of felt like the anti-Witness. The Witness was fastidiously fair and so carefully constructed I can look back at it and marvel, but actually playing it was pretty formal and mirthless. Baba is You can be a little sloppy and unfair, but it's warm and fun and funny, and the actual craftsmanship of the puzzles themselves is still top tier, in it's own way.
That's Windows-only, though. Someone recently released a "demake" with fewer levels for Pico-8 that you can play in a browser: https://www.lexaloffle.com/bbs/?tid=142638
A game jam is like a weekend hacking event where a bunch of people make small games in a short time frame and then get feedback or compared. A demake is a remake but usually for older systems or in an older style / lower resolution. In the indie game scene lots of the big hit games were first created at a game jam. No tomfoolery, they're useful words to describe things, that don't have other easy ways to describe.
Demakes can also be games that boil down the essence of their genre or original game to simpler rules. A great example of this is Footsies, which is a demake of the trad fighting genre.
Game jam - like a hackathon, for games. Groups are given a theme or conditions and a short time (eg a weekend) to make a game as a challenge / for fun.
Demake - remaking a game for a more limited set of hardware. Remake normally implies increasing quality, whereas a demake is focused on cramming as much quality as possible in a much more constrained execution environment. (In this case, a PC/iOS game to a Pico-8 game.)
Demake in modern times is often used for example with porting DOOM to (insert electronic device like a temperature sensor that shouldn't normally play games). Since the target platform has less features, it's not a re-make, it's a de-make. This could also be for more academic purposes, like "what kind of game would we make if we deleted all the guns out of DOOM".
But it comes from the old days of gaming. You could play an amazing fighting game at the arcade, but the home console had considerably less power. So the home version of that game might be very limited.
Back then we would just call that a port. Port was used to mean taking something from its native platform and putting it on something else, which was rarely an upgrade. Putting an NES/famicom game onto Genesis/master system would involve upgrades (sprite capability) and side or downgrades (sound chip differences).
Port today generally means a game has the same quality across all platforms, barring the bare facts of hardware capability. So de-make now often means a port with some intentional limitation. "I know a digital home pregnancy test can't run DOOM, but can it run enough to be playable?"
DOOM on a pregnancy test is still a port and not a demake.
A demake is not specifically on more primitive hardware but is in a morr primitive style that may run on more primitive hardware.
But it is from dialogue that knowledge emerges. Please feel free to ignore questions that don't interest you. I appreciate the question and its answers.
I bought it on recommendation from an earlier HN thread, and my 7-year-old, 8-year-old and I play it together. We're only about three hours into the game, but it has been very enjoyable. As a sibling comment says, the puzzles are often very funny to all three of us, and even the youngest of us is able to suggest and find solutions. There have been levels where the 7-year-old thought of the solution before I did.
Also available on Google Play Store [1], and it successfully derailed my sleep plans last night. Really breaks your expectations of "I move the character around" within minutes.
It reminds me a bit of the card game Fluxx [2], where every card you play changes the rules.
When playing something like UNO we did the same, whoever wins one game can add any rule for the next and all previous rules are kept, unless if it turns out that a rule we added ruins the game, then we remove it.
After that it becomes a mad scramble to remember all the previous rules while coming up with new wacky additions.
Humans have the ability to inspect anything from any point of view that we want, like a real time video game simulation system in our minds.
I suspect that artificial neural networks are missing this core ability to encode and translate its point of view when processing data. Just look at basic LLMs failing at the reversal curse: https://arxiv.org/abs/2309.12288 No amount of more GPUs and more training data will fix it.
The entorhinal cortex in humans has a special kind of neuron called grid cells that works with hippocampus cells to do generalized coordinate translations. The discovery of grid cells won a nobel peace prize ( https://www.nobelprize.org/prizes/medicine/2014/advanced-inf... ). It is a prime suspect to incorporate for a coordinate transformation layer in my opinion.
I want to build an ANN that incorporates this "coordinate transformation system", but I'm not sure how to translate it into code, this is my dream project.
The problem is an ANN or LLM has no physical location or “point of view”, which has led ML practitioners toward the concept of embodiment: where algorithms and agents no longer learn from datasets of images, videos or text curated primarily from the internet - instead they learn through interactions with their environments from an egocentric perception similar to humans.
Curiously I didn’t see any mention of this term in your paper. But you did touch on the contrast between egocentric and allocentric representations, which lies at the heart of the issue.
I think the grand challenge of AGI will be unifying these two to reap the benefits of both while minimizing the dangers of imbalance. A bit like physics wrt gravity and quantum mechanics.
We’ll need allocentric perspective for globally optimal decisions (superorganism), but we’ll also need egocentric reasoning to enable robust autonomy in unique local environments.
Also worth mentioning that multimodal representation learning seems like it essentially achieves the sort of cross modality behavior of grid cells, although physical location (not positional encoding) is not an input in any of the research I’m aware of.
Are there groups actively attempting this? I was thinking that this type of data gathering is a simple solution to LLMs “running out of data” to train on. Just put a camera in a room and let the model learn by exploring its environment.
You’re looking for group equivariance - there is some work in group equivariant neural networks. You might find the very recent paper on Clifford Group-Equivariant networks interesting - it applies to not only Euclidean spacetime transformations but even Minkowskian spacetime.
You might also be interested in implementations for novel view synthesis such as instantngp and other NERF-type models. InstantNGP is actually ludicrously simple relative to most recent neural architectures, imo.
With respect to the reversal problem, I wonder how strongly this would still hold for eg encoder-decoder (eg T5) or “order-free autoregressive” models (eg sigma-GPT from last week)
Positional embeddings used in transformers are actually suspiciously similar to grid cells. They allow for a very similar kind of coordinate transform.
You might enjoy this paper[0] which shows that recurrent position encodings recover grid cell representations and maps to path integration found in a popular model of the hippocampus. This isn't terribly surprising since RNNs have shown this before[1, 2] but its an interesting connection.
The game is truley mind blowing. The rules are simple and clear, there are no special hidden tricks, yet almost every puzzle goes from "it's impossible" to "oh didn't think of that". Truley shatters you assumptions over and over.
While on the topic of games that change your thinking or give you skills, any recommendations?
A few I benefited from:
Chess: for general lessons of resourcefulness, creativity, calculation, pausing before acting, and practicing being comfortable with irresolution (e.g. when two or more pieces could be traded but you have to resist the urge to simplify if doing so worsens your position). Chess also opened my eyes to how much distractions and inadequate sleep affect cognition (my elo can drop 100 points when badly lacking sleep, and another 100 if playing in a public place with distractions).
Vim Adventures: not exciting enough to recommend for gaming alone, but it made it bearable to repeat vim key strokes for 3-4 hours per day for a few days straight until I had the muscle memory to use it for most coding tasks without too much clumsiness.
DuoLingo: less of a 'game'; more of an educational tool, but still worth the mention as it made learning foreign language much easier than doing so via book/audio.
It sets you up with a logistical mindset, you learn to think about setting up a "supply chain" before you do something. This becomes a very abstract skillset that you start to apply in multiple areas of your life.
Secondly, it takes your thinking to a higher, more abstract level when you learn to build factories that build factories that build...
Thirdly, you learn polymorphic thinking in a high level. Like, products that are then taken through different processes produce entirely different things, and then some of them are fed back to this system.
I have played the game for low two-digit hours, but it augmented my thinking in a concrete way.
I will highly recommend Factorio.
Chess has also benefited me in being more far-sighted in shorter terms. Especially in situations with real rivals, or even without.
Scrabble had helped me learn in an early age that clever tricks and pragmatism help me win more than qualities like "elegance", etc. It was a lesson 'thrown in my face'.
I have noticed the same effect from studying different methods of information organization (such as ontologies or mind maps).
Experience with a structured mental path is a fundamental part of learning. I've seen many educational games that explicitly teach skills, but don't necessarily offer an experience path for the application of those skills. Factorio is more along the lines of what an educational game should be.
I actively collect methods of information organization, so "which have I studied" is too broad to list in a comment.
Learning resources - the greatest learning resource is teaching yourself how to learn, and understanding how your personal learning system works.
When I go to learn something:
- I don't use just one source. I look at multiple sources, comparing different perspectives as part of the learning process. No one author will explain things in "just the right way" to make things click.
- I build a dynamic MECE (mutually exclusive, collectively exhaustive) "framework" as I learn a subject, which can organize every piece of information I might need to save. If a piece of information does not fit the framework, the framework is changed.
- I use image search to quickly find different perspectives of a particular concept.
Information organization recommendations: In general, I would start with "the basics" - a process - even if you're sure you understand it well. See if you can organize what you know into a MECE framework, and find out if your understanding of the information is "siloed" (based on a condition, such as context). Ask yourself how broadly you are able to apply your understanding, or pick an "unrelated" subject and organize it from a process view as an exercise. Look at different models of a process and which components are always included, and which are not.
Factorio is amazing and the way it makes you remap certain parts of your brain works differently depending on the person, and the paths they took in the game, so it is a little hard to describe.
Based on how other people try to explain it, I'd say the closest comparison matches the experience of being a "traditional" programmer being exposed to haskell.
You've kind of just got to play it to experience it yourself.
There's two parts to factorio:
1. You have to figure out what the next goal you need to focus on is, and then work backwards through the steps you'd need to achieve to get there.
2. You need to wrap your head around the task at hand figuring out how to build pipelined systems in a limited space. You might think you need A which can be turned into B which can be turned into C, which is correct. But it turns out the next step you planned requires some of your C to be turned into D while still making C, then you need to combine come D with B to make E, but suddenly you're not getting enough C to make D because you're using too much of the B to make E. You can swap some of the things that turns A into B to make B at a faster rate, but the A to B thing takes up more physical space, so now you need to move the things around in space without ending up getting tangled.
Repeat these two things in a loop for the basic game loop, then add in the occasional emergencies that you didn't foresee that can wildly change your current objective.
The difficulty curve in the game is designed in such a way to always let you figure out what your next objective is, but without you knowing all the pieces to get there.
And there are multiple paths to the end so you need to figure out how you're getting there.
Plus each of the ways you can use to get to the end work in different ways, sometimes subtle, sometimes utterly different.
Because Factorio is always training your brain in different ways, you'll find that you're way of problem solving has become better every 10 hours or so. You'll randomly get ideas out of the game of how to improve things or do things in a different way and you can try them out whenever you get back to the game.
Even when you finish it, you can go back and retry it to see just how much you've retrained your brain. You might even want to try the different paths to brain train other methods.
You can finish the game for the first time in about 50 hours. Depending on how much you game, this may be a lot, or hardly anything, but I'll point out that it's not one of those repetitive daily-mission grind-fest second job games. It's much more like reading a really good book series that you're enjoying.
Play CTFs, maybe the Flare-On challenge if you particularly enjoy reversing. Microcorruption is a just small taste of what the full CTF scene has to offer. picoCTF and pwn college are good starting points for beginners.
while True: learn() comes to mind. Solving puzzles that involve thinking in terms of flows, accidentally learning some machine learning, and things that would transfer to tasks like designing computer networks, or scaling apps where the challenge is some choke point.
Polytopia is chess-like to me, in that there are many scenarios where you need to move precisely in order to pull off an attack or defense. While chess would be considered to be heavily focused on military and territory, Polytopia requires you to balance military, territory, economy, and technology development. It generates a lot of questions for me, that I end up trying to answer with small coding projects, like what’s the optimal way to expand using this tribe on this map type, or whatever.
Helps with reflexes and hand eye coordination but more importantly it fosters a zen mindset. Have to roll with the punches and work the problem instead of getting frustrated after you envitably die for the 1000th time.
In general transferring skills through games is quite hard, except for the skill you are actually practicing in the game (i.e. by playing chess you will mostly improve your chess ELO). There is some research going on to broaden this area though [0].
Where gaming and more recently the 'educational' apps one sees advertised on certain websites and youtube channels definitely do help is raising awareness and motivation to the featured scientific disciplines.
From experience, I can say that there are many games with transferable skills.
At the top of my list is Factorio, and second is actually World of Warcraft.
More generally, there is a lot to learn from game design that you can learn through playing. The question of how to make taxes palatable, for example, is really no different to the question of how to challenge a player without frustrating them.
I believe the value of games are poorly understood, partially because chess, a game that has nothing left to teach us, muddies the conversation
> "because chess, a game that has nothing left to teach us"
That's a wild and weird take. It may not teach the collective "us", if your "us" is a very small set of well-established, experienced, smart, and mature people always in their element, then, yes, Chess has arguably nothing left to teach "us".
But it can be a very good learning experience for the absolute majority of people.
What about Go, or Checkers, or Call of Duty, or...?
That's the issue. Yes sitting down and concentrating on difficult tasks for long periods of time is good exercise. So is jogging, which isn't sitting down. Or rowing. Or lifting weights. Or...
Chess probably has much less benefits than learning a musical instrument, for example.
I will say that playing counter strike for money has given me confidence in stressful situations in a workplace, as well as helping me navigate tension between team members/co-workers.
It is mostly soft skill type stuff. For example, something goes wrong and veers from an original plan, and you have to come up with a solution and adapt in the moment.
An idle game like cookie clicker can teach you much about what addiction could look like for you. It depends a lot on the person what kind of game will "click" and if they will "click" at all, but if they do you will be utterly addicted for a while.
This sounds like an awful idea!
I have an addictive personality, and cookie clicker and similar idle games have taught me a lot about how I feel when I'm addicted to something, how it affects my decision making capabilities, how I can detect it early, what I can do to get out of the loop.
This is not something you'll pick up automatically by playing these kinds of games, you'll have to deliberately and mindfully interact with them, but if you do you can learn a lot about how you respond to addictive stimuli and hopefully prevent a lot of way, way worse addiction in the future through these lessons.
I recommend the vim adventures demo for the gaming alone. Haven't tried the rest of it, having balked at the price tag (knowing vim well enough that I think I'd be paying for the gaming alone).
Portal 2 has some truly unbelievable custom maps that hit the sweet spot for brain games mixed with hi-if graphics IMO. If you’ve never played the low-grav lunar levels or used the time traveling gun, you’re missing out.
Into the Breach is the closest you’re ever going to get to a chess sequel.
I love the Talos Principle 1 and 2, if you liked Portal you will probably like them, they rely less on platforming and more on puzzles that require you to use the puzzle elements in creative ways. The puzzle elements are easy to understand, like a jammer beam gun which jams a force field door when placed. One of the early and obvious tricks is if you have two jammers, how you can bring them both with you through a door.
They also have quite a bit of humanist philosophy, in my opinion they do a good job of being relentlessly optimistic in a way that feels genuine without being overbearingly techno-optimist.
DuoLingo is deceptive at best. It's made addictive at the outset so you feel good. You feel like you're learning. But long term the platform has what is now dubbed enshittification. It gets exponentially harder to keep up with reviews, there are known problems with the lesson structure that will never be addressed, etc. And if you pay money, they'll make it easier for you. Hm.
Real critique example: Duo relies on 1:1 translation. Native language learning isn't even close to 1:1. Every language learner hits a point where they can say, "I know exactly what this phrase means in the target language, and there is no direct translation in English". To finish a language in Duo, you will have to memorize dozens/hundreds+ of totally misleading and bad translations like this.
Tldr. Selling kids DuoLingo for language learning is like selling kids cigarettes to help them learn to build a fire.
Few moments (or hours, or days, depending when you are reading this post) ago a custom level set, titled "The Legend of Zelbaba: Linkeke between words" was presented @party 2024 - https://scenesat.com/video/atparty . Actual presentation starts at around 17-ish minute of live stream. Level pack is here (base game required): https://gamebanana.com/mods/150971
Shameless plug for a side project i have been working on for a few years, but if you like Baba is You you may enjoy Pathology or Sokopath. They can be played on my site Thinky.gg (https://thinky.gg)
Theres a pretty active community on Thinky publishing block pushing levels every day.
Levels range from Kindergarten to Super Grandmaster in difficulty. Also features a level editor where you can publish custom levels to the world.
>> Among other things, it dissolves barriers between data and algorithm, between a cognitive system and its contents, and gets you to think differently.
Ah, I remember my epiphany, 13 ish years ago when Prolog first clicked for me. I put it into a little rhyming couplet:
There is no separation
'tween data and operation
Funny how you have to go all the way to the highest level just to find the same simple truths that define the lowest level; of computation.
> And the new factor comes into being, when one sees, when the mind realises totally that the observer is the observed, when that realisation takes place there is a release of energy which is new, which will go beyond 'what is'.
—Jiddu Krishanmurti, Public talk in Rome, 1973 October 28.
Fun fact, the developer behind this game also made another technical marvel of a game called Noita. A game best described by the fact most players don't realise they've only played the tutorial.
It seems similar to the hiring process for Jane Street, as described in "Going Infinite", by Michael Lewis. Jane Street tests applicants on how well they do with games where the rules change during the game.
Baba Is You would be perfect for teaching people security, penetration testing, and so on.
Lateral thinking puzzles are underrated, and orgs mostly punish lateral thinkers. Even though the lateral thinker is going to catch way more bugs. They will also generate more bug reports by doing so, and that's bad for numbers, stop that.
Or the old FANG hiring problem. "How would you implement a binary search tree for this problem, and do it only on a Google doc, no auto complete."
Good programmer: "I would never do that. That sounds horrible. Who thought of this?"
I love a good lateral thinking puzzle, but in my experience too many of them wind up being "read the author's mind" guessing games and misleadingly-worded "gotcha" silliness instead of actual exercises in thinking about a problem.
Have you played some of the pure puzzle video games out there? Many of them involve meta layers and secrets that involve lateral thinking and epiphany to progress in.
You have sequences of symbols, with attached mechanics. Arrange the blocks in different order, to get different outcomes.
Maybe I'm just grumpy today, I don't see how the game is "profound".
edit: I guess what I have in mind is if anything, the fact we can create meaning out of any collection of symbols, and decide waht it means, shows that language is really empty and intelligence is completely outside of language. Language is just a tool.
I'm not sure what you mean. How would you replace the words with icons and still have the exact same mechanics, and how would having the exact same mechanics be different if it were icons instead of words?
I just installed Baba is You. It reminds me of Sokoban. I and my friend spent several hours playing Sokoban in college. It was frustratingly difficult.
> There’s also plenty of opportunity to involve AI in this kind of virtual world.
interesting type of game, even though i would not personally play it. however, the author shared a resource to simulate the game for reinforcement learning (https://github.com/utilForever/baba-is-auto).
not many languages go that far as to allow to redefine what "call" is/does, for example. Though, some do.
"Baba is you" is pretty cool, it is not only about changing rules, it also at some point plays with what language is, and you can even stretch it to explore some platonic ideas.
The game this article is based on is “Baba Is You” [1]. It’s a logic puzzle game where you try to get “you” to “flag” — or more precisely, “win” (which, by default most of the time, “flag is win”) — but you manipulate a number of objects and join them with other words to change the “rules” of the game.
I highly recommend it to the HN crowd.
[1] https://store.steampowered.com/app/736260/Baba_Is_You/
Something that set Baba is You apart for me compared to any other puzzle game I can remember playing is the puzzles themselves were actually "funny." Not like Portal, where you solved puzzles alongside a funny narrative, but in the actually language of the puzzles themselves there are setups which get subverted in absurd and delightful ways, often in multiple layers, as you work your way through to a solution. Playing made me feel like the math part of my brain was laughing.
The game doesn't hold your hand, and I think it took about 10 hours for me to learn enough of the puzzle language for things to get really good (and then it started to descend into frustrating fiddliness in the deep endgame), but the middle 30 hours or so we're some of the most gratifying gameplay I've experienced.
It kind of felt like the anti-Witness. The Witness was fastidiously fair and so carefully constructed I can look back at it and marvel, but actually playing it was pretty formal and mirthless. Baba is You can be a little sloppy and unfair, but it's warm and fun and funny, and the actual craftsmanship of the puzzles themselves is still top tier, in it's own way.
If anyone wants to try some levels before buying it, the game jam version of the game is on itch.io: https://hempuli.itch.io/baba-is-you
That's Windows-only, though. Someone recently released a "demake" with fewer levels for Pico-8 that you can play in a browser: https://www.lexaloffle.com/bbs/?tid=142638
What is a game jam? What is a demake?
A game jam is like a weekend hacking event where a bunch of people make small games in a short time frame and then get feedback or compared. A demake is a remake but usually for older systems or in an older style / lower resolution. In the indie game scene lots of the big hit games were first created at a game jam. No tomfoolery, they're useful words to describe things, that don't have other easy ways to describe.
Demakes can also be games that boil down the essence of their genre or original game to simpler rules. A great example of this is Footsies, which is a demake of the trad fighting genre.
https://store.steampowered.com/app/1344740/FOOTSIES_Rollback...
Game jam - like a hackathon, for games. Groups are given a theme or conditions and a short time (eg a weekend) to make a game as a challenge / for fun.
Demake - remaking a game for a more limited set of hardware. Remake normally implies increasing quality, whereas a demake is focused on cramming as much quality as possible in a much more constrained execution environment. (In this case, a PC/iOS game to a Pico-8 game.)
Demake in modern times is often used for example with porting DOOM to (insert electronic device like a temperature sensor that shouldn't normally play games). Since the target platform has less features, it's not a re-make, it's a de-make. This could also be for more academic purposes, like "what kind of game would we make if we deleted all the guns out of DOOM".
But it comes from the old days of gaming. You could play an amazing fighting game at the arcade, but the home console had considerably less power. So the home version of that game might be very limited.
Back then we would just call that a port. Port was used to mean taking something from its native platform and putting it on something else, which was rarely an upgrade. Putting an NES/famicom game onto Genesis/master system would involve upgrades (sprite capability) and side or downgrades (sound chip differences).
Port today generally means a game has the same quality across all platforms, barring the bare facts of hardware capability. So de-make now often means a port with some intentional limitation. "I know a digital home pregnancy test can't run DOOM, but can it run enough to be playable?"
DOOM on a pregnancy test is still a port and not a demake. A demake is not specifically on more primitive hardware but is in a morr primitive style that may run on more primitive hardware.
Search engines are your friend :)
But it is from dialogue that knowledge emerges. Please feel free to ignore questions that don't interest you. I appreciate the question and its answers.
Over the years more and more people would disagree with this though
Enjoy it while it lasts…
I bought it on recommendation from an earlier HN thread, and my 7-year-old, 8-year-old and I play it together. We're only about three hours into the game, but it has been very enjoyable. As a sibling comment says, the puzzles are often very funny to all three of us, and even the youngest of us is able to suggest and find solutions. There have been levels where the 7-year-old thought of the solution before I did.
Also available on Google Play Store [1], and it successfully derailed my sleep plans last night. Really breaks your expectations of "I move the character around" within minutes.
It reminds me a bit of the card game Fluxx [2], where every card you play changes the rules.
[1] https://play.google.com/store/apps/details?id=org.hempuli.ba...
[2] https://www.looneylabs.com/games/fluxx
Baba Is You strikes me as an exceptional game for phones, since one can do a single level at a time and the controls are simple. Thanks for the link.
I own it and play it and it challenges the limits of my thinking. I feel I can recommend it despite the fact that I myself get stuck.
Has a lot of “aha” moments.
I love being able to change the rules of a game.
When playing something like UNO we did the same, whoever wins one game can add any rule for the next and all previous rules are kept, unless if it turns out that a rule we added ruins the game, then we remove it.
After that it becomes a mad scramble to remember all the previous rules while coming up with new wacky additions.
This game would be a nice AI (note, there's no "G") benchmark
I've written about the cognitive aspect the author talks about before:
https://blog.jtoy.net/examining-concepts-through-different-f...
https://blog.jtoy.net/on-the-wondrous-human-simulation-engin...
https://arxiv.org/abs/2210.12068
The human brain has at least over 12 different coordinate systems. I like to reference this image: https://cln.sh/gpqhwrw2Hg92MQz62TY6
Humans have the ability to inspect anything from any point of view that we want, like a real time video game simulation system in our minds. I suspect that artificial neural networks are missing this core ability to encode and translate its point of view when processing data. Just look at basic LLMs failing at the reversal curse: https://arxiv.org/abs/2309.12288 No amount of more GPUs and more training data will fix it.
The entorhinal cortex in humans has a special kind of neuron called grid cells that works with hippocampus cells to do generalized coordinate translations. The discovery of grid cells won a nobel peace prize ( https://www.nobelprize.org/prizes/medicine/2014/advanced-inf... ). It is a prime suspect to incorporate for a coordinate transformation layer in my opinion.
I want to build an ANN that incorporates this "coordinate transformation system", but I'm not sure how to translate it into code, this is my dream project.
The problem is an ANN or LLM has no physical location or “point of view”, which has led ML practitioners toward the concept of embodiment: where algorithms and agents no longer learn from datasets of images, videos or text curated primarily from the internet - instead they learn through interactions with their environments from an egocentric perception similar to humans.
Curiously I didn’t see any mention of this term in your paper. But you did touch on the contrast between egocentric and allocentric representations, which lies at the heart of the issue.
I think the grand challenge of AGI will be unifying these two to reap the benefits of both while minimizing the dangers of imbalance. A bit like physics wrt gravity and quantum mechanics.
We’ll need allocentric perspective for globally optimal decisions (superorganism), but we’ll also need egocentric reasoning to enable robust autonomy in unique local environments.
Also worth mentioning that multimodal representation learning seems like it essentially achieves the sort of cross modality behavior of grid cells, although physical location (not positional encoding) is not an input in any of the research I’m aware of.
Are there groups actively attempting this? I was thinking that this type of data gathering is a simple solution to LLMs “running out of data” to train on. Just put a camera in a room and let the model learn by exploring its environment.
You’re looking for group equivariance - there is some work in group equivariant neural networks. You might find the very recent paper on Clifford Group-Equivariant networks interesting - it applies to not only Euclidean spacetime transformations but even Minkowskian spacetime.
You might also be interested in implementations for novel view synthesis such as instantngp and other NERF-type models. InstantNGP is actually ludicrously simple relative to most recent neural architectures, imo.
With respect to the reversal problem, I wonder how strongly this would still hold for eg encoder-decoder (eg T5) or “order-free autoregressive” models (eg sigma-GPT from last week)
Positional embeddings used in transformers are actually suspiciously similar to grid cells. They allow for a very similar kind of coordinate transform.
You might enjoy this paper[0] which shows that recurrent position encodings recover grid cell representations and maps to path integration found in a popular model of the hippocampus. This isn't terribly surprising since RNNs have shown this before[1, 2] but its an interesting connection.
[0] https://arxiv.org/abs/2112.04035
[1] https://arxiv.org/abs/1803.07770
[2] https://www.nature.com/articles/s41586-018-0102-6
The game is truley mind blowing. The rules are simple and clear, there are no special hidden tricks, yet almost every puzzle goes from "it's impossible" to "oh didn't think of that". Truley shatters you assumptions over and over.
While on the topic of games that change your thinking or give you skills, any recommendations?
A few I benefited from:
Chess: for general lessons of resourcefulness, creativity, calculation, pausing before acting, and practicing being comfortable with irresolution (e.g. when two or more pieces could be traded but you have to resist the urge to simplify if doing so worsens your position). Chess also opened my eyes to how much distractions and inadequate sleep affect cognition (my elo can drop 100 points when badly lacking sleep, and another 100 if playing in a public place with distractions).
Vim Adventures: not exciting enough to recommend for gaming alone, but it made it bearable to repeat vim key strokes for 3-4 hours per day for a few days straight until I had the muscle memory to use it for most coding tasks without too much clumsiness.
DuoLingo: less of a 'game'; more of an educational tool, but still worth the mention as it made learning foreign language much easier than doing so via book/audio.
I have truly benefited from playing Factorio.
It sets you up with a logistical mindset, you learn to think about setting up a "supply chain" before you do something. This becomes a very abstract skillset that you start to apply in multiple areas of your life.
Secondly, it takes your thinking to a higher, more abstract level when you learn to build factories that build factories that build...
Thirdly, you learn polymorphic thinking in a high level. Like, products that are then taken through different processes produce entirely different things, and then some of them are fed back to this system.
I have played the game for low two-digit hours, but it augmented my thinking in a concrete way.
I will highly recommend Factorio.
Chess has also benefited me in being more far-sighted in shorter terms. Especially in situations with real rivals, or even without.
Scrabble had helped me learn in an early age that clever tricks and pragmatism help me win more than qualities like "elegance", etc. It was a lesson 'thrown in my face'.
I have noticed the same effect from studying different methods of information organization (such as ontologies or mind maps).
Experience with a structured mental path is a fundamental part of learning. I've seen many educational games that explicitly teach skills, but don't necessarily offer an experience path for the application of those skills. Factorio is more along the lines of what an educational game should be.
What are the methods that you have studied? What learning resources did you use?
What are some of the methods that you would recommended?
I actively collect methods of information organization, so "which have I studied" is too broad to list in a comment.
Learning resources - the greatest learning resource is teaching yourself how to learn, and understanding how your personal learning system works.
When I go to learn something:
- I don't use just one source. I look at multiple sources, comparing different perspectives as part of the learning process. No one author will explain things in "just the right way" to make things click.
- I build a dynamic MECE (mutually exclusive, collectively exhaustive) "framework" as I learn a subject, which can organize every piece of information I might need to save. If a piece of information does not fit the framework, the framework is changed.
- I use image search to quickly find different perspectives of a particular concept.
Information organization recommendations: In general, I would start with "the basics" - a process - even if you're sure you understand it well. See if you can organize what you know into a MECE framework, and find out if your understanding of the information is "siloed" (based on a condition, such as context). Ask yourself how broadly you are able to apply your understanding, or pick an "unrelated" subject and organize it from a process view as an exercise. Look at different models of a process and which components are always included, and which are not.
Tony Zhu has a whole video about Factorio and software engineering: https://youtu.be/vPdUjLqC15Q
Can you give specific examples for how you’ve applied skills learned in Factorio to real life situations?
Factorio is amazing and the way it makes you remap certain parts of your brain works differently depending on the person, and the paths they took in the game, so it is a little hard to describe.
Based on how other people try to explain it, I'd say the closest comparison matches the experience of being a "traditional" programmer being exposed to haskell.
You've kind of just got to play it to experience it yourself.
There's two parts to factorio:
1. You have to figure out what the next goal you need to focus on is, and then work backwards through the steps you'd need to achieve to get there.
2. You need to wrap your head around the task at hand figuring out how to build pipelined systems in a limited space. You might think you need A which can be turned into B which can be turned into C, which is correct. But it turns out the next step you planned requires some of your C to be turned into D while still making C, then you need to combine come D with B to make E, but suddenly you're not getting enough C to make D because you're using too much of the B to make E. You can swap some of the things that turns A into B to make B at a faster rate, but the A to B thing takes up more physical space, so now you need to move the things around in space without ending up getting tangled.
Repeat these two things in a loop for the basic game loop, then add in the occasional emergencies that you didn't foresee that can wildly change your current objective.
The difficulty curve in the game is designed in such a way to always let you figure out what your next objective is, but without you knowing all the pieces to get there.
And there are multiple paths to the end so you need to figure out how you're getting there.
Plus each of the ways you can use to get to the end work in different ways, sometimes subtle, sometimes utterly different.
Because Factorio is always training your brain in different ways, you'll find that you're way of problem solving has become better every 10 hours or so. You'll randomly get ideas out of the game of how to improve things or do things in a different way and you can try them out whenever you get back to the game.
Even when you finish it, you can go back and retry it to see just how much you've retrained your brain. You might even want to try the different paths to brain train other methods.
You can finish the game for the first time in about 50 hours. Depending on how much you game, this may be a lot, or hardly anything, but I'll point out that it's not one of those repetitive daily-mission grind-fest second job games. It's much more like reading a really good book series that you're enjoying.
Please use your imagination. My comment is abstract, and deliberately so. You can apply these in any field of life.
http://microcorruption.com if you want to get into reverse engineering.
What do you do after you beat it?
Play CTFs, maybe the Flare-On challenge if you particularly enjoy reversing. Microcorruption is a just small taste of what the full CTF scene has to offer. picoCTF and pwn college are good starting points for beginners.
while True: learn() comes to mind. Solving puzzles that involve thinking in terms of flows, accidentally learning some machine learning, and things that would transfer to tasks like designing computer networks, or scaling apps where the challenge is some choke point.
Polytopia is chess-like to me, in that there are many scenarios where you need to move precisely in order to pull off an attack or defense. While chess would be considered to be heavily focused on military and territory, Polytopia requires you to balance military, territory, economy, and technology development. It generates a lot of questions for me, that I end up trying to answer with small coding projects, like what’s the optimal way to expand using this tribe on this map type, or whatever.
I'm going to put a plug in for Elden Ring.
Helps with reflexes and hand eye coordination but more importantly it fosters a zen mindset. Have to roll with the punches and work the problem instead of getting frustrated after you envitably die for the 1000th time.
Believe it or not Dead Cells got me personally organized by highlighting the importance of good tool strategies with limited carry slots
In general transferring skills through games is quite hard, except for the skill you are actually practicing in the game (i.e. by playing chess you will mostly improve your chess ELO). There is some research going on to broaden this area though [0].
Where gaming and more recently the 'educational' apps one sees advertised on certain websites and youtube channels definitely do help is raising awareness and motivation to the featured scientific disciplines.
[0] - https://link.springer.com/content/pdf/10.1007/s41465-022-002...
From experience, I can say that there are many games with transferable skills.
At the top of my list is Factorio, and second is actually World of Warcraft.
More generally, there is a lot to learn from game design that you can learn through playing. The question of how to make taxes palatable, for example, is really no different to the question of how to challenge a player without frustrating them.
I believe the value of games are poorly understood, partially because chess, a game that has nothing left to teach us, muddies the conversation
> "because chess, a game that has nothing left to teach us"
That's a wild and weird take. It may not teach the collective "us", if your "us" is a very small set of well-established, experienced, smart, and mature people always in their element, then, yes, Chess has arguably nothing left to teach "us".
But it can be a very good learning experience for the absolute majority of people.
>(i.e. by playing chess you will mostly improve your chess ELO)
I have found it to help with strategy, especially multi-round strategy. This is useful in business and other areas of life, not just chess ELO.
Opportunity cost.
What about Go, or Checkers, or Call of Duty, or...?
That's the issue. Yes sitting down and concentrating on difficult tasks for long periods of time is good exercise. So is jogging, which isn't sitting down. Or rowing. Or lifting weights. Or...
Chess probably has much less benefits than learning a musical instrument, for example.
I will say that playing counter strike for money has given me confidence in stressful situations in a workplace, as well as helping me navigate tension between team members/co-workers.
It is mostly soft skill type stuff. For example, something goes wrong and veers from an original plan, and you have to come up with a solution and adapt in the moment.
https://online-go.com/learn-to-play-go
An idle game like cookie clicker can teach you much about what addiction could look like for you. It depends a lot on the person what kind of game will "click" and if they will "click" at all, but if they do you will be utterly addicted for a while.
This sounds like an awful idea!
I have an addictive personality, and cookie clicker and similar idle games have taught me a lot about how I feel when I'm addicted to something, how it affects my decision making capabilities, how I can detect it early, what I can do to get out of the loop.
This is not something you'll pick up automatically by playing these kinds of games, you'll have to deliberately and mindfully interact with them, but if you do you can learn a lot about how you respond to addictive stimuli and hopefully prevent a lot of way, way worse addiction in the future through these lessons.
I recommend the vim adventures demo for the gaming alone. Haven't tried the rest of it, having balked at the price tag (knowing vim well enough that I think I'd be paying for the gaming alone).
Portal 2 has some truly unbelievable custom maps that hit the sweet spot for brain games mixed with hi-if graphics IMO. If you’ve never played the low-grav lunar levels or used the time traveling gun, you’re missing out.
Into the Breach is the closest you’re ever going to get to a chess sequel.
I love the Talos Principle 1 and 2, if you liked Portal you will probably like them, they rely less on platforming and more on puzzles that require you to use the puzzle elements in creative ways. The puzzle elements are easy to understand, like a jammer beam gun which jams a force field door when placed. One of the early and obvious tricks is if you have two jammers, how you can bring them both with you through a door.
They also have quite a bit of humanist philosophy, in my opinion they do a good job of being relentlessly optimistic in a way that feels genuine without being overbearingly techno-optimist.
DuoLingo is deceptive at best. It's made addictive at the outset so you feel good. You feel like you're learning. But long term the platform has what is now dubbed enshittification. It gets exponentially harder to keep up with reviews, there are known problems with the lesson structure that will never be addressed, etc. And if you pay money, they'll make it easier for you. Hm.
Real critique example: Duo relies on 1:1 translation. Native language learning isn't even close to 1:1. Every language learner hits a point where they can say, "I know exactly what this phrase means in the target language, and there is no direct translation in English". To finish a language in Duo, you will have to memorize dozens/hundreds+ of totally misleading and bad translations like this.
Tldr. Selling kids DuoLingo for language learning is like selling kids cigarettes to help them learn to build a fire.
Few moments (or hours, or days, depending when you are reading this post) ago a custom level set, titled "The Legend of Zelbaba: Linkeke between words" was presented @party 2024 - https://scenesat.com/video/atparty . Actual presentation starts at around 17-ish minute of live stream. Level pack is here (base game required): https://gamebanana.com/mods/150971
Shameless plug for a side project i have been working on for a few years, but if you like Baba is You you may enjoy Pathology or Sokopath. They can be played on my site Thinky.gg (https://thinky.gg)
Theres a pretty active community on Thinky publishing block pushing levels every day. Levels range from Kindergarten to Super Grandmaster in difficulty. Also features a level editor where you can publish custom levels to the world.
>> Among other things, it dissolves barriers between data and algorithm, between a cognitive system and its contents, and gets you to think differently.
Ah, I remember my epiphany, 13 ish years ago when Prolog first clicked for me. I put it into a little rhyming couplet:
Funny how you have to go all the way to the highest level just to find the same simple truths that define the lowest level; of computation.Games are cool too.
> And the new factor comes into being, when one sees, when the mind realises totally that the observer is the observed, when that realisation takes place there is a release of energy which is new, which will go beyond 'what is'.
—Jiddu Krishanmurti, Public talk in Rome, 1973 October 28.
https://www.krishnamurti.org/transcript/when-the-mind-realis...
This typically comes with Lisp or Haskell, prolog is an interesting case!
Fun fact, the developer behind this game also made another technical marvel of a game called Noita. A game best described by the fact most players don't realise they've only played the tutorial.
Baba is you is similarly a good introduction to the mindset needed to write exploits.
It seems similar to the hiring process for Jane Street, as described in "Going Infinite", by Michael Lewis. Jane Street tests applicants on how well they do with games where the rules change during the game.
Baba Is You would be perfect for teaching people security, penetration testing, and so on.
Lateral thinking puzzles are underrated, and orgs mostly punish lateral thinkers. Even though the lateral thinker is going to catch way more bugs. They will also generate more bug reports by doing so, and that's bad for numbers, stop that.
Or the old FANG hiring problem. "How would you implement a binary search tree for this problem, and do it only on a Google doc, no auto complete."
Good programmer: "I would never do that. That sounds horrible. Who thought of this?"
I love a good lateral thinking puzzle, but in my experience too many of them wind up being "read the author's mind" guessing games and misleadingly-worded "gotcha" silliness instead of actual exercises in thinking about a problem.
Have you played some of the pure puzzle video games out there? Many of them involve meta layers and secrets that involve lateral thinking and epiphany to progress in.
If you replace "WALL" "IS" "STOP" with icons/images, it's not that interesting anymore imho.
At the very least, Captain Blood would come to mind.
https://www.abandonware-france.org/ltf_abandon/ltf_galeries....
You have sequences of symbols, with attached mechanics. Arrange the blocks in different order, to get different outcomes.
Maybe I'm just grumpy today, I don't see how the game is "profound".
edit: I guess what I have in mind is if anything, the fact we can create meaning out of any collection of symbols, and decide waht it means, shows that language is really empty and intelligence is completely outside of language. Language is just a tool.
I'm not sure what you mean. How would you replace the words with icons and still have the exact same mechanics, and how would having the exact same mechanics be different if it were icons instead of words?
I just installed Baba is You. It reminds me of Sokoban. I and my friend spent several hours playing Sokoban in college. It was frustratingly difficult.
Related albeit tiny:
Baba Is You - https://news.ycombinator.com/item?id=25560525 - Dec 2020 (1 comment)
More:
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
> There’s also plenty of opportunity to involve AI in this kind of virtual world.
interesting type of game, even though i would not personally play it. however, the author shared a resource to simulate the game for reinforcement learning (https://github.com/utilForever/baba-is-auto).
Just learn a programming language that allows for metaprogramming.
not many languages go that far as to allow to redefine what "call" is/does, for example. Though, some do.
"Baba is you" is pretty cool, it is not only about changing rules, it also at some point plays with what language is, and you can even stretch it to explore some platonic ideas.
> not many languages go that far as to allow to redefine what "call" is/does, for example. Though, some do.
Scheme would be an example that also allows for metaprogramming.
with the c pre processor, you can do anything.
but, please don't.
Or Objective-C method swizzling