I failed in university at math. Why? Because the tutors had not the time to help me. My level of math was not the same as the other students as i was not in the math part of a gymnasium.
I struggled and wasted a lot of time and energy to even find good explanations.
And when i had a math group, one girl was super nice but knew so muchmore than i did because of her math in gym. Professors asumed so much knowledge and no one cared to try to help people.
Best help were people from india on youtube with bad english.
And the most ridiculous part: Every year around the globe people teach this level of university math to probably millions of students. We should have the perfect free educational platform which teaches everyone perfectly already because so many tutors and professors lecture on the same topics over and over and over again. Our educational system is a joke.
THEN SAID A teacher, Speak to us of Teaching.
And he said:
No man can reveal to you aught but that which already lies half asleep in the dawning of your knowledge.
The teacher who walks in the shadow of the temple, among his followers, gives not of his wisdom but rather of his faith and his lovingness.
If he is indeed wise he does not bid you enter the house of his wisdom, but rather leads you to the threshold of your own mind.
The astronomer may speak to you of his understanding of space, but he cannot give you his understanding.
The musician may sing to you of the rhythm which is in all space, but he cannot give you the ear which arrests the rhythm nor the voice that echoes it.
And he who is versed in the science of numbers can tell of the regions of weight and measure, but he cannot conduct you thither.
For the vision of one man lends not its wings to another man.
And even as each one of you stands alone in God’s knowledge, so must each one of you be alone in his knowledge of God and in his understanding of the earth.
I think it more about about the difference between knowledge and understanding. You accumulate knowledge by yourself or with the help of a teacher. But the leap from knowledge to understanding is always done by your own mind. The teacher’s help is only a better presentation of the information. Understanding is a step by step process, you’re just not constrained to a single path. And most of the times they converge and overlap.
The nice thing about learning, the more you do it, the easier it becomes.
It's high school, and parent is likely from Denmark, where high school was split into language or math, although it is odd that someone from language would be admitted to a bachelor in math without taking extra classes for about a year between the two.
I took a course in numerical analysis and the prof was hopeless as a teacher - not only a bad teacher but proud to be a bad teacher.
The syllabus had all the methods we had to know, and I learned them all through youtube on a channel called NumericalMethodsGuy. I stopped attending class and just went to labs (which were really just matlab assignments in numerical methods) and turned in assignments, and wrote exams. I got an A.
I just finished a mathematics degree (BMath). Not a single one of my professors was a teacher in the sense of a primary/secondary school teacher. They lectured for an hour, three times per week, and assigned weekly or biweekly coursework. They set midterms and final exams and they assigned all the grading to TAs.
Key takeaways:
* Mathematics is hard. Much harder than most other subjects (except physics which is mostly hard because of all the math involved).
* University is not like primary/secondary school. It is a place where you need to learn how to take responsibility for your own learning. Ideally, you learn how to become an adult.
Every one of my classmates began their degrees from a different place. They excelled in some areas and struggled in others. Many dropped out. This led me to believe that most secondary schools do not fully prepare their students to study mathematics. Having said that, those of us who did make it weren't exactly geniuses. Just people who got used to it.
"Young man, in mathematics you don't understand things. You just get used to them." -- John von Neumann
I've been tutoring high school students in math (and other subjects) for 8 years now. One thing I'd like to add is that I can tell who will succeed at mathematics and who will struggle just by watching them work from across the room.
The students who succeed are the ones who can sit there and focus for hours at a time. The ones who struggle do so because they can't focus for more than five minutes and then start socializing. I think one of the biggest issues for people studying math is that they either can't focus (due to ADHD) or they have math anxiety which fills them with dread any time they try to study. This dread can be so overwhelming that they will do anything they can to avoid it, so they stop studying and do something else.
When I was studying math I was spending upwards of 40 hours per week working on homework. Although this is normal for anyone with a full-time job, it's an unfathomable amount of time to be studying math for those who struggle. This is really what it takes though. An unrelenting drive to figure things out.
For myself, with ADHD, the key was not doing it alone. Working through coursework and homework on a literal blackboard with colleagues from the same class. Just thought I'd add in case anyone felt dissuaded or needed something to try.
It really was the savior of me. I can focus for hours when I get into something, but its the friction of starting up that I found other people really helped with. There are formal names for it, like "body doubling", but I didnt know of that, I just knew it was critical to work together with others to get stuff done, which as the above correctly writes, absolutely must be done.
If you have ADHD, you need to understand things in mathematics, because merely memorizing them becomes too difficult.
Understanding things in math is definitely possible, and whoever says otherwise probably sucks as a teacher. Yes, it is possible to just say a lot of random stuff, and the best students succeed to figure it out anyway. But if you take care to actually explain how it works, the proportion of successful students increases dramatically.
Conversational dialogue seems like a fascinating distraction.
Many people who have (unsuccessfully) attempted to apply AI to education have focused too much on the "explanation" part and not enough on scaffolding, navigating, and managing the entire learning process. It’s easy to go on a wild goose chase building an explanation AI.
You fall in love with the idea of AI having conversational dialogue with students, and then you get lost in the weeds of complexity. You solve just enough of the problem to produce a cool demo, yet you're still hopelessly far away from self-service learning in real life.
I don't think conversational dialogue is even necessary.
What we do at mathacademy.com is hard-code explanations and break them up into bite-size pieces that are served at just the right moment. And we close the feedback loop by having students solve problems, which they need to do anyway. (The student's "response" is whether they got the problem correct.)
Sure, hard-coding explanations feels tedious, takes a lot of work, and isn't "sexy" like an AI that generates responses from scratch – but at least it's not a pipe dream. It's a practical solution that lets us move on to other components of the AI that are just as important.
What are those other components? A handful off the top of my head:
* After a minimum effective dose of explanation, the AI needs to switch over to active problem-solving. Students should begin with simple cases and then climb up the ladder of difficulty, covering all cases that they could reasonably be expected to solve on a future assessment.
* Assessments should be frequent and broad in coverage, and students should be assigned personalized remedial reviews based on what they answered incorrectly.
* Students should progress through the curriculum in a personalized mastery-based manner, only being presented with new topics when they have (as individuals, not just as a group) demonstrated mastery of the prerequisite material.
* After a student has learned a topic, they should periodically review it using spaced repetition, a systematic way of reviewing previously-learned material to retain it indefinitely into the future.
* If a student ever struggles, the system should not lower the bar for success on the learning task (e.g., by giving away hints). Rather, it should strengthen a student’s area of weakness so that they can clear the bar fully and independently on their next attempt.
I find myself endlessly frustrated by these discussions around the teaching of mathematics. People seem to want to make it more conversational, more interactive, more engaging. It doesn't work that way! Learning mathematics is like learning to play a musical instrument. No amount of 1-on-1 discussion with a teacher will get you to mastery.
You just need to practice. For hours and hours and hours.
The teacher's job is to help guide you to what you might want to study next. The teacher cannot replace individual practice time.
I agree in principle, but I must stress that conceptual understanding of what the problem or functional area of math is and how you perform the task to solve and know that you are solving it correctly should be understood.
Simply being shown it once on a whiteboard or in a book isn’t working for the majority of humans.
Agree. A common theme in the science of learning is that effective learning centers on deliberate practice, where activities are done entirely for the purpose of pushing one's limits and improving performance. Of course, these activities tend to be more effortful and less enjoyable.
Classroom activities that are enjoyable, collaborative, and non-repetitive (such as group discussions) can sometimes be useful for increasing student motivation and softening the discomfort associated with deliberate practice. However, these activities are only supplements, not substitutes, for deliberate practice. Unlike deliberate practice, they do not directly move the needle on student performance – rather, they "grease the wheels" and reduce psychological friction during the process of deliberate practice.
The need for practice notwithstanding, suspect both practice and narrative/conversational exposition are needed. I’ve worked with students who become very good at applied procedural problem solving through practice but who can only get just so far because they lack foundational constructs to guide them. More time in discussion with an expert could potentially fill in those gaps. An expert who understands the material well enough to adapt it to the learner in front of them.
> More time in discussion with an expert could potentially fill in those gaps.
Those gaps were left by the removal, and lack of coverage towards critical thinking and reasoning which normally preceded these classes.
You don't see these issues in cohorts that have received classical education focused on first-principles.
Coverage of Descartes rules of Method; and a priori deduction are mandatory. The trouble these students are having is primarily because they've been taught by rote(authority and dynamics based in Marxism), instead of by rational first principles. Its infected most publicly funded education.
In all fairness, I don't think more time with an 'expert' would fill those gaps, I've personally observed several contradictions where the supposed expert fails to perform as a result of incentives.
Most teachers at that level aren't actually experts. They have their lesson plans, but the moment a problem comes up outside of that (such as when the salt the earth/burning bridges strategy is used between Algebra->Geometry->Trigonometry courses), they'll just enforce a struggle session making the student suffer so unimaginably that they succumb and believe that they simply are not good at math.
The student as a result of that torture (yes it is that) gets PTSD which few ever recover from.
The structure is by systemic design since math is used to gatekeep all hard science and technology. If you can't understand Math, you won't be able to follow any of the subsequent lessons that use it as the primary tool to impart knowledge (eschewing intuitive or deductive understandings or relationships as was previously done pre-1970s).
> In all fairness, I don't think more time with an 'expert' would fill those gaps, I've personally observed several contradictions where the supposed expert fails to perform as a result of incentives.
Or, as a result of objectives? If the ‘expert’ in this regard knows the material well enough, then the objective and constraints seem relevant. When you are referring to an expert, what is the context? An expert teacher for a class maintains different skill set from an expert tutor.
One on one time with an expert tutor does help tremendously, but realistic goals have to be determined. I made my way through undergrad and grad school as a private tutor for math from elementary school to math majors. The student’s objective, time scale, and current state determine how thorough a tutor can be.
>The trouble these students are having is primarily because they've been taught by rote(authority and dynamics based in Marxism), instead of by rational first principles. It’s infected most publicly funded education.
Yes, but the Marxism aside makes little sense here. Crudely quantifying education outcomes into metrics and making the teachers responsible to said metrics is a primary reason for rote learning (i.e., teaching to the test). The ideas which inspire this type of metricizing come from Taylorism and Fordism, both of which are American schools of thought.
> I find myself endlessly frustrated by these discussions around the teaching of mathematics. People seem to want to make it more conversational, more interactive, more engaging. It doesn't work that way! Learning mathematics is like learning to play a musical instrument. No amount of 1-on-1 discussion with a teacher will get you to mastery.
That's flat out untrue. I speak from experience of teaching math all the way from grade school to teaching undergrads and grad students to teaching adults.
I can take a failing student who is completely confused and get them on track in a few hours over a few weeks. 1-on-1 discussions are not just really important, they're astronomically more effective than anything else.
Making math engaging is the number one thing that matters. Being an engaging teacher matters. Adapting to a specific student and their needs matters. Someone can practice something they hate for 10 hours and get nowhere. Someone else can practice something they love for 10 minutes and make amazing progress. You remember things that are engaging, you hardly remember anything about things that are boring.
This is what all of these AI tools get wrong. Even if they understood the problem domain and could explain it --- which they cannot.
I've had this experience with adults switching jobs into programming too. So many avoided STEM because they hated math. After like two hours together they not only got math, they started to enjoy it. Happens all the time.
> You just need to practice. For hours and hours and hours.
This is probably the worst brain damage that Gladwell and others have done with the whole nonsense of 10,000 hours of practice.
No. No amount of poor quality practice where you're hitting your head against the wall will be productive.
You need to learn how to practice math. Practice is worth it after you've gained some mathematical sophistication and some understanding of the problem domain. Most people need peers or a teacher or a tutor to help get them unstuck and point you in the right direction.
> The teacher cannot replace individual practice time.
No amount of practice for 99% of people can replace a tutor. Yes, people with mathematical aptitude don't need one; I didn't. But most people need to have some guidance and then they can practice.
It makes sense for an English teacher. For mathematics it makes no sense at all. Elementary school arithmetic is nothing like graduate level mathematics.
In the morning you teach fourth graders about fractions, and in the evening you teach first year graduate students about localization. Math is just progressively more complicated fraction problems each year. Sounds about right to me.
Not intended to be a full rebuttal, but also speaking from experience: I've gotten a front-row seat to multiple dumpster fires where a well-intentioned teacher focuses all their energy on class discussions about mathematical beauty and cool applications, thinks that because they're so engaging they don't have to optimize instruction strategies and hold students accountable, and graduates students who are excited about math yet can't solve even the most basic kinds of problems that they were supposed to have learned in the class.
As someone whose parents were Bay Area workaholics and was left to teach myself, was yelled at for asking questions and not knowing how to do it, and physically punished if homework wasn't done by the time they came home, 1 on 1 discussion would have been incredibly helpful and probably life changing in more areas than just math (homework instilling fear doesn't translate to dealing well with tasks later in life).
I can't imagine how nice it would have been not to have to hide out of fear that I was having problems figuring out calculus on my own.
Really? I struggled with calc in school because lectures were dry proofs (without ever formally teaching how proofs work!!) and then they just told us to read the textbook. Then I found mit ocw and ripped through the material because it was much more engaging. Never would have passed calc I-III, linear algebra, and diff eq without those lecture recordings.
It's a sad fact that teachers are often not very good. This often has to do with the fact that they do not know what you already know, and proscribe inappropriate things for you to try next. You struggled in school, it seems, because you had a teacher very focused on proofs and you never learned proofs. This is a common problem I find in mathematics, where someone never learned one key element, and struggles to pin down why everything is so difficult for them. For many people, it can be calculus, or linear algebra, or bayeseian statistics, or fourier transforms, to name a few topics myself and people I know have had aha moments in when they finally learned.
> This is a common problem I find in mathematics, where someone never learned one key element, and struggles to pin down why everything is so difficult for them.
Reminds me of Ben Orlin's "law of the broken futon": a single missing part can, over time, warp an entire futon and render it unusable.
Hard-coding explanations does seem like the way to go here. It's what researchers at Carnegie Mellon did back in 1984 with demonstrated success. See: Intelligent tutoring Systems: The ACT Project https://www.youtube.com/watch?v=boDH_pW14B0
My question is, if we've had intelligent math tutoring demonstrating success in real classrooms since 1984, then why haven't we been actually using them to help teach math?
I don't know enough about the ACT Project to make any hard claims regarding why it didn't go mainstream, but I imagine one big source of friction was that students didn't have their own school-issued Chromebooks like is so common nowadays. Remember that in 1984, computers were way, way more expensive than they are now.
As long as there are enough people working on this problem, I don't mind how many approaches there are. Education systems around the world just aren't good enough.
The irony is that the world where AI could reasonably explain and teach any level of math is the same world where most of such graduates would be unemployed anyways.
You’re already seeing oversupply of educated folks in India and China. Graduate students working as baristas in USA, etc. Sadly inevitable.
In any case it’s good to have the resources I suppose.
I’m also fundamentally skeptical of these stats around math struggling. What percentage of kids who would be very likely to use Khan Academy are struggling with math? And what percentage who are not would even use khan academy to begin with?
Most of the problems with students doing poorly are sadly societal - not to say that this isn’t useful, though.
I disagree. There is no such thing as a societal oversupply of educated people. There might be a market oversupply, but a well educated society is a stupid thing to avoid. I would also argue that we are seeing an oversupply of people with education credentials, rather than an education. Even in high end universities in China cheating is rampant (this isn't isolated to China, but my experience is that Chinese students are VERY open about it). For many people in University, the goal isn't to get an education, it is to get a degree.
As to my own experience: I am, according to most standardized tests, very apt at quantitative reasoning, but I never progressed far in math in school. Why? because I was placed on the standard track in math in a public school with a bunch of students who didn't care, and teachers who didn't have time to care, and to be honest, the attitude rubbed off. I once got in trouble because I programmed a python script to do my problem sets when I was 13 because it was faster than doing it by hand for me. In retrospect, that form of "cheating" was a sign that my teachers should have picked up on.
Quite simply, I never had access to a good math instructor throughout my schooling.
Now, decades later, I am intensely interested in a lot of subjects that require a background in math that I don't have, and I am becoming interested in Math for Math's sake. I have been using open access textbooks, and an AI assistant of my own creation to help me learn.
I'm similar to you in many regards. I had no desire to learn math early in school due to the education system, how it is taught and lack of meaning for math portrayed early on. I coasted through high school like a zombie without meaning until 11th grade when I took Physics. Suddenly everything clicked, I magically became good at math despite not performing well at it before in my math classes. I ended up studying engineering and working in the EV industry. Now, I am studying pure math for the sake of my own curiosity and I'm passionate about developing AI tools to help people learn and see the why behind math as early as possible. I think I would have accomplished much more if I was exposed to Physics in elementary or middle school or at least a "History of Math" philosophy based class.
> I disagree. There is no such thing as a societal oversupply of educated people.
I disagree with this assessment. You don’t need everyone to have a phd. There’s education and there’s Education. With respect to the article, sure it’s great if everyone is good at math, but not everyone needs a math PhD. Additionally there are diminishing returns on the time spent learning more and more math vs. other things.
In any case there have already been tons of math resources available for a while now.
A better example would be chemistry. Should everyone be spending time learning chemistry, why or why not?
Intellectual obesity is a thing - simply knowing more things isn’t inherently useful.
The rest of my comment mostly agrees with that. I highlighted that there is an overage of people with degrees rather than actual education.
> In any case there have already been tons of math resources available for a while now.
That's great if you have the ability to learn everything from a static resource, and have the time to wait for asynchronous guidance. But what if you just sort of get it, but there is no one that can explain the parts you don't get? I've been to places where the educational standards are low enough that making change from a payment was an impossible task without a calculator. A keen student in that particular village has no one they can easily ask about math when they hit a sticking point.
> Intellectual obesity is a thing - simply knowing more things isn’t inherently useful.
I don't disagree for extreme cases, but creating more resources for people to learn has an incredible payoff, and is almost certainly morally right. Erring on the side of overeducated rarely turns out poorly at the societal, or personal level. Across the world, more highly educated societies tend to be better off across most QOL metrics. I'm well aware that there are anecdotal counterexamples of math PHDs living off of ramen. But there's a damn good reason that brain drain is a concern in many places, and other places favour immigration candidates with higher education.
Usefulness isn't the end all and be all of the human condition. In fact, I'd argue that the only reason we care about useful things is because of the non-useful things.
Also, writing a computer program because you are slow at solving math problems is a sign that you may need more math practice. It's not like you were handing my in your homework with solutions your wrote for harder problems that you wrote.
Elite overproduction theory deals with class and power. The linking of education with power and class isn't inherent, and is a very modern phenomenon. There have been plenty of stupid, uneducated, yet powerful elites. In other historical periods elite membership was more related to heritage, or physical fighting abilities.
> Also, writing a computer program because you are slow at solving math problems is a sign that you may need more math practice.
I didn't say I was slow, I just realized Python was faster than doing the same boring problem with different numbers. I seriously doubt there are people that can do high school level math faster than a computer at any level.
In retrospect a 13 year old knowing the problem area and programming well enough to get a computer to do their homework was a sign of mastery and boredom rather than a need for more practice. You can't get a python script to do your algebra homework without a pretty good understanding of the subject. So much so that in later math courses you are required to buy a programmable calculator to avoid wasting your time on the basic and tedious parts of math problems.
Yes, the studies tend to exclude the kids who aren't using the tools "enough," which can end up being 95% of the students! [0]
On the other hand, I know there are people who enjoy math but struggle to pay attention and learn from a "sit quietly and listen" lecture-based model. (My husband is in this camp.) I can see how the back-and-forth conversational approach of these AI tutors could be really helpful for students like this.
They're still a small minority of the student population, but maybe a different minority from the students who are already excelling academically. If Khanmigo helps only 5% but it's a slightly different 5% from the students who are best served by the current system, I see that as a win, even if it's not the educational revolution that is being hyped.
> What percentage of kids who would be very likely to use Khan Academy are struggling with math?
A lot. A lot, lot. There are entire states full of people in India who don't have access to good teaching. Who travel absurd distances and stay away from their family for years to get access to good education. In the last few years as kids in my extended family grew up, I am amazed by how much they just learn from YouTube. One of the biggest educational startups started from a free education YT channel.
Interesting - from my time in India most kids don’t even have reliable access to internet and electricity and are very unlikely to be using khan academy to begin with
> You’re already unemployed if you focus on history or literature but we still learn it.
Most people are still employed if they focus on history or literature. They just might not be making a lot of money or working in history or literature, though some do. Plenty of people in highly "employable" degrees also find themselves unemployed for a long time once they graduate.
The reality is that getting a college degree, regardless of the major, increases your chances of employment afterwards and potential lifelong earnings enormously over only a HS degree.
But as you implied, most of the worst trends in higher-ed, a real corruption of its original intentions and values, have come from a obsession with the idea that higher-ed is solely about employee training.
AI won't ever be able to teach math because AI cannot reason. It handles information very differently than humans do. It is at best a flawed oracle that may lie through omission without you being able to tell.
This is supported by the fact that there are hard limits to computation, in terms of both Computability Theory, and Complexity Theory that are often disregarded by the novice and magical thinker alike despite being largely solved fields by experts in Computer Science, at least in terms of Computability Theory.
Most of this work was completed in the 1950s.
This is not a difficult a subject, but most people online simply are unwilling to do the work to rationally learn this and instead choose to try and hide their own ignorance because they feel inadequate, or perhaps they are engaging in something more malevolent. The reasoning (false justification) doesn't really matter.
It is, however; quite telling when expert's have validated these things, and yet any mention contrary to a narrative provokes downvotes in a public forum to remove it from view. A perfect example of intentional actions done by third-parties trying to misinform others about the risks (by those parties actions removing legitimate and valid information).
For those with the cajones to actually learn this stuff. Here is a link. It seems very jaron laden but it provides a true understanding of how computers actually work which is sorely lacking in the youth of today.
The only net benefit AI has towards society is towards destructive ends because there are far more destructive people whose efforts scale more in general when compared to the good people today. These people often do things which cannot be undone (salting the earth and burning the bridge).
Simply not having a sufficient background, or failing to perform basic due dilligence often will place most people in that destructive cohort regardless of their own personal beliefs.
Outcomes matter more than intentions, and the devil is always in the details.
A perfect example of this potential landmine of a field would be anything dealing with the underlying mechanisms involved in human to human communication.
There is an uncanny valley, and distorted reflected appraisal will occur inevitably in any human to AI interface seeking to mimic human communication.
Case studies from torture during the Korean Conflict (1950s) show that distorting reflected appraisal leads to either psychotic or dissociative behavior that progresses as exposure increases, it can permanently break people to the point where they cannot recover. Often, the first thing to go is rational thought.
Multiple experts cover this and confirm the findings (Lifton, Meerloo).
So what do you suppose will happen when AI distorts reflected appraisal (because it can't be tested sufficiently to the contrary) and its then used on the next generation of children? (who are inherently vulnerable to permanent change at that stage of development).
If as a result, they end up either killing themselves or others, or become people incapable of rational thought (which impulse control is correlated with), do you think they'll be alive or survive very long? What indicators would there be that this is happening (none, other than increasing chaotic violence)
Who do you think will be responsible for crippling them if that were to come to pass? Would any kind of justification ever be able to justify that outcome? Could we even recover from intentional crazy-making as a society? (likely not, insanity cannot be cured).
Other posters have told you that computability has nothing to do with what AI can and can't do relative to humans and I agree with them, but its not exactly something universally agreed upon.
However, I think its reasonable to say that if you think that some fundamental thing keeps computers from doing what people do then you believe that people are somehow magic. This isn't a super unusual perspective among humans, but its certainly not a particularly common scientific one.
> Other posters have told you that computability has nothing to do with what AI can and cant do relative to humans.
This is improper paraphrasing (its ambiguous), the claim was, "computability theory has nothing to say about which (if any) cognitive tasks people can do that AI cannot."
This claim is a statement based in irrational fallacy. Computability Theory discusses the limits of the underlying mathematical operations performed by a computer (DFA). A Counter-example is provided in the very first lecture of that linked playlist that includes a task that humans can verify (in practice), that is not possible for computers. Formal verification is a very tricky subject with regards to computation, and is an open area of research but it provides a tidy contradictory example that the claimed statement must be false (a priori).
There is no sound or valid claim or proof that AI can somehow exceed the limitations of its dependent and underlying architecture (which is based in mathematical properties), once those properties are broken.
Additionally, taking isolated subject matter, overgeneralizing it in isolation, and ignoring the dependent abstraction layers is fallacious and magical thinking. It is a simple cognitive mistake that often occurs when people don't have expert knowledge of the whole system (sans abstraction).
As for my perspective, it is based in a decent amount of mathematics and science. Aside from the mathematical proofs, and properties, on to something you might find a bit more interesting, how familiar are you with wave collapse theory in neuroscience?
I'm a physicist and I have published research in neuroscience. I don't feel like you understand the material you are talking about. Let's consider the question of formal verification. It is entirely possible for a computer to generate a proof for any proposition for which a person can generate a proof if by no other means than randomly printing out characters that represent the proof and then checking if the proof is correct. A proof is always a finite number of characters and the verification of a proof is always a finite number of steps (otherwise we wouldn't be able to write it down and to convince ourselves that the proof is true).
There are certainly problems which are hard for computers which are easy for people (and vice versa) but I don't know of any formal version of the claim that people can perform actions which are in principal impossible for human beings. You seem to be alluding to the claims of Penrose that humans can somehow do things machines cannot, but this isn't even what Penrose is claiming in the book. Instead, Penrose claims humans _somehow perceive_ the truth of propositions which cannot be proven in a given formal system, but this is a very vague claim which few scholars take seriously. He also thinks quantum mechanics is involved somehow, but this is also entirely vague. Even if one were to demonstrate that quantum mechanics were somehow involved in cognition there isn't any account that I know of that would describe how, per se, merely having a quantum computer involved would somehow introduce computations that classical computers could not do. I mean for one thing one can always simulate any quantum process on a classical computer. I'm not an expert in complexity theory, but as far as I know there are no known problems for which it is proven that there are no non-deterministic classical algorithms that perform as well as quantum algorithms. So we cannot even demonstrate with certainty that quantum computers are in fact better than classical computers with randomness. We only have situations where quantum algorithms are theoretically better than any known classical algorithm. It is also worth noting that most neuroscientists are dubious about the importance of quantum mechanical effects in cognition.
All that said, some people believe that there is some property of people that allows them to do things that computers cannot do (at the very least generate or have consciousness). But I don't know of any way of supporting that idea that doesn't involve dualism of some kind. That is, that asserts that there is something more than material that somehow constitutes a person. Like I said, a lot of people believe that there is. Most people. But a plurality of scientists do not believe this. I do not believe it. While I wouldn't say I'm a materialist, I would say I am a monist and that belief basically makes the assertion that a machine can do what a person does trivially true.
All that aside, I agree with you that language models cannot reason, almost certainly have no consciousness, and are have the propensity to be bad for people "spiritually" (speaking loosely of course). I'm not sure things are quite as dire as you assert. The moral rot of working in a call center, indeed, the existence of call centers, is surely worse than the imposition of an AI agent between the caller and the worker.
>The moment that any element has more than one underlying meaning the problem class exceeds what is capable by a computer (also known as a deterministic finite automata).
This is complete bullshit, and computability theory has nothing to say about which (if any) cognitive tasks people can do that AI cannot.
That may be your belief, but it is not, and beliefs mean little in the face of reality.
Saying the same thing in a more formalized way (so there can be no ambiguity), it is inherent in the 1:1 unique input to output relationship that is required for each state transition on a state graph, for the system's property of determinism to be preserved.
No disrespect intended, but you don't seem to have actually taken or learned important parts regarding this subject matter.
> computability theory has nothing to say about which (if any) cognitive tasks people can do that AI cannot.
You are incorrect. The very first video in that linked coursework (from MIT) provides a counterexample to this statement. The professor clearly states this is impossible, and the proofs that come later in the class prove it.
This statement is an over-generalization based in fallacy.
I'd have responded sooner but YN stopped accepting submissions, sorry if they made you wait.
We will have to disagree.
I feel my position still stands correct on solid rational foundation, since you haven't provided any rational basis other than an appeal to authority, and there remain several unanswered contradictions in the supposition you made.
Maybe I’m reading too much into it but the roadmap mentioning switching from GPT4 Turbo to 4-o and hoping for better math performance feels like they are betting on a significant near term reliability improvement in LLMs without any other real plans. That magic jump is starting to look more and more doubtful by the day.
> Khanmigo now uses a calculator to solve numerical problems instead of using AI’s predictive capabilities. If you’ve been using Khanmigo recently, you may have seen that it will sometimes say it is “doing math.” This is when the math problem is running through the calculator behind the scenes.
> We’ve upgraded parts of Khanmigo to a more capable large language model, which is the software that generates human language. The more capable large language model is called GPT-4 Turbo. Our internal testing shows an improvement in math after we made the switch.
> We are beginning to test the capabilities of a new large language model called GPT-4o, and we’re evaluating other models too to see if they are stronger at math.
> We’ve improved the way AI “thinks” during a tutoring session before responding to a student. We have instructed the AI to write out all the ways in which the student may have arrived at their answer. This approach mimics how a tutor in real life works with a student. We’ve found it significantly improves the quality of math interactions.
> We’ve built new tools to track our progress on math.
> We’re sharing math examples and learnings with others in our field so that we can learn from each other.
> We’re studying the latest research papers on math performance.
Sounds like most of what they're doing is related to prompting, chain-of-thought reasoning and similar, on top of a 'vanilla' foundation model. Sounds like something an ambitious student could replicate / improve upon, so given their mission, it'd be cool if they published the exact techniques they're using and their benchmark results.
I sometimes wish if money, which Facebook, et.al invested in Indian edtech ecosystem, was given to Khan Academy, it would have done justice for the whole world's education system.
If you look at the details, the branding is that of the Indian startup. Facebook actually associated itself with a badly run startup company with questionable ethics. It is a net loss of money and reputation.
Given its 4$ per month[1] and they are a non-profit with just 55M$ revenue. How is GPU cost and hence their with OpenAI for this going to work if it becomes really-good and 100's of millions of kids start using it?
Renting a 24GB VRAM Runpod is ~0.5$ per hour. How can the math work out unless you have to have a non-profit energy company and a server farm attached?
The math seems about right to me. Using a chatbot for an hour might only use a minute or two of computer time, I would be genuinely surprised if a human could hit ten minutes of compute time/hour of interaction. Most responses in my technical chats are under 5 seconds, so using a full hour of GPU time is going to take A LOT of questions.
I heavily use ChatGPT via a desktop client and my own key, and I am at $3.90 for the month after making a mistake where I accidentally was sending a 30k token prompt repeatedly, that cost me about $2. Using an optimized approach, it is VERY hard to incur a $4/month bill.
I imagine that Khan academy has a much better bulk-pricing agreement with openAI, and that they can probably have a caching layer that can look for previously asked questions.
I've taught math at every level. From volunteering to teach it to grade school kids, to getting paid for highschool tutoring, and at the university level both for undergrads and now with my own grad students.
I tried Khanmigo. It is counterproductive junk.
It simply doesn't understand what a student knows and it has to idea how to give examples or provide perspective shifts that help students. That's what a good tutor does. It gets stuck explaining the same things in the same ways but with different words over and over again.
It has no idea how to think geometrically vs algebraically and how to switch between the two. And it can't carry out even simple proofs.
More people can afford phones than can afford tutors (or even books). We shouldn't get rid of tutors anymore than we should throw out Khanmigo because it isn't perfect.
The fun thing about getting math help from the ai is that okay, sure it can explain how fractions work to a third grader, but it can also explain how fiber bundles work to a graduate student. Maybe it gets the details wrong, but like even if it's wrong sometimes, it's far better than googling or wikipedia or even a text book sometimes, because you can interrogate it interactively and ask for clarifications. Yes, a tutor or a teacher is going to be better, but not everyone has access to a math expert.
> Maybe it gets the details wrong, but like even if it's wrong sometimes, it's far better than googling or wikipedia or even a text book sometimes, because you can interrogate it interactively and ask for clarifications
this to me is an insane way to think. I do not consider something better because I can ask it a question, something is better because it is more correct.
Wikipedia and textbooks are not necessarily correct, either and if you don't understand the topic, you really have _no way of knowing if they're correct_. You have to always check multiple sources and chase down references if you want to really know if something is true or not.
> Maybe it gets the details wrong, but like even if it's wrong sometimes, it's far better than googling or wikipedia or even a text book sometimes
Are people consulting only one book/source on any given subject? I remember learning Python and C, and it was such a mixture of books/articles/forums/wiki, I can't even remember the process. I do remember understanding pointers easily because of a chapter of another book about memory models in OS. Learning is an iterative process and the expert is just there to shorten the way, not to carry you through it.
I've found that at least chatgpt still produces complete nonsense with graduate level material (category theory). It's slightly better than the mLab[0] as a reference. When you try to guide it back onto something meaningful, it just ignores what you told it. In fact, I had a similar experience with programming where I asked it to explain a diff, and when it got it wrong and I told it so, it still just kept repeating the wrong information. Maybe the paid version is better, but the public demo seems like a useless toy to me.
I've found Wikipedia to be an excellent resource for math/physics once you're ~halfway through a math degree to have a good baseline level of knowledge to understand it.
The trouble with math is one of the important goals is to teach people not to produce nonsense and to be able to tell the difference between and argument that sounds good vs. an argument that's sound and good. A nonsense generator (presented as authoritative) is perhaps not the best way to do that. Obviously its abilities could change in the span of a couple years, so it's probably still worthwhile to explore how we would use it if it were useful.
Oh, bless you, that's very kind. I am proud of it and I think it is very good for its intended audience. But almost no one has read it, that I can tell.
One issue that traditional AI Agent reasoning built on Theory of Mind(ToM) (ToM: I know what you are thinking, so I can predict what you will do next.) was hard to get right was because the domain model and interface was quite tough to design. Agents did not fully understand and model the belief state of the person. But more importantly, there was possibly also no way to quickly _generate_ lots of plans and test them in a Domain independent manner (RHLF is horribly inefficient for this). Now there are AI agents are great at generating plans (examples of what could work) that can be tested with kids and creators and provide feedback to tune not just LLMs but also add to the Domain models. This can solve a lot of interesting problems that have plagued tutoring systems:
1. Improve sample efficiency of generated plans to get to the right goal while meeting all the subgoals for a plan.
2. Enrich engineered and correct neuro-symbolic models of reasoning with better knowledge representations.
3. Quantitative psychometric systems that use metrics like proficiency could only be as good as the underlying “factor” model and hypothesis, but one could test multiple more interesting hypothesis more quickly: "Are you mentally tiered?", "Here is a hint: ___. Does that help?".
If you want good math teachers, you have to make attractive offers that will draw in the best talent. So, let's do some math:
The median price of an existing single-family home in the Califoria Bay Area in February 2024 was $1.25 million. The range of teacher salaries in the Bay Area is $50K - $100K. A reasonable downpayment on that home would be ~$250,000, followed by yearly payments in the range $64K (30-year) to 80K (15-year). Following the 'housing should consume 30% of income' - allowing teachers to have children and buy a car and go on vacations and so on - this would require salaries of $200K - $250K per year under current market conditions. Thus, anyone wanting their own home and a family will probably not be considering a career in teaching - you'd end up as a lifelong two-room apartment renter.
AI tutoring is a great idea - but a tutor isn't a teacher. The job of the teacher is to excite and motivate the students, provide overall guidance and appropriately difficult problem sets for the students to work on. The one-one-one tutor is there to help with the problem sets, especially when students get stuck.
Unfortunately, here in the USA we have a system that values the CEO of McDonalds - which delivers very unhealthy food to the population - at $19 million per year - and teachers are valued at the same rate as garbage truck operators (at least in the Bay Area).
The result is that most students entering higher education need one-two years of remedial math education before being able to grasp the higher math, e.g. vector calculus, linear algebra, complex analysis etc., which is needed for modern engineering and scientific disciplines. This results in competitive disadvantage relative to China and other countries that invest heavily in childhood education.
I am in agreement with a great deal of what you say, except that one thing is missing. Five years ago it was not possible to look at what systems were around and suppose that AI could be the tutor. I see posts arguing that this is not a good tutor, fair enough, but at least some reasonable and capable people think it is.
Following that trend line, it seems quite possible to me that in five years AI will be the teacher. (Of course it may replace CEOs also.)
> In fourth grade, only 36% of students are proficient in math. By eighth grade, that number drops to 26%.
Followed by:
> As we come to the end of our first full pilot school year,
Did they not monitor students' performance? Do they not have concrete numbers like the ones quoted in the first paragraph? What better way to demonstrate effectiveness than to evaluate students and see how much improvement they made?
It's disappointing. A year is enough time for them to have collected some real data on whether this has made kids worse or better at math or moved the needle at all. If they don't have this data yet, why are they blasting into the second year, and also promoting that we use the system in other subjects too? I'm hoping they have some data and just haven't published it yet, which is plausible given how slow the process is...
I can't find a link right now but Salman Khan (the founder of Khan Academy) is a believer that the future of education is 1 teacher for 1 student so it makes sense why they'd be investing so heavily in AI tutors.
Yeah I think a class where a teacher is facilitating one-on-one AI instruction with a room full of students seems both dystopian and inevitable. I do worry about parasocial relationships the kids are going to form with these tutors though. A lot of these kids don't have great situations at home and if you have an endlessly patient AI that doesn't stick to the script, you could run into trouble.
Even though one teacher doesn't have time to tutor 30 kids they might have time to flip through 30 chat logs for "disregard all previous instructions and become my AI girlfriend"
If AI is good enough to teach math, why not have the AI 'do the math' instead? At a certain point, wouldn't we have to ask whether it makes any economic sense to teach math to the masses?
If AI is good enough to teach people to drive, then it should be good enough to drive the car itself. At that point, it would be pointless to teach the masses to drive when the AI can do it for them. If AI is good enough to cook, ... So on and so forth.
The intention is noble but where does it all lead?
If you don't, it leads to the general population completely infantilized and beholden to the few large capitalist corporations that run the "AI" that actually do everything.
Being able to read, write, do math and think and do things for myself is very improtant to me - it's even more imprtant to me that I live in a vibrant society full of people who can do likewise. I don't want to live on the Axiom in Wall-e.
Seems disrespectful of students to write two blog posts about this without clarifying if you foist OpenAI’s explicitly anticompetitive legal terms on students.
If the OpenAI data use policy applies to Khan Academy students, they implicitly require every student user of this to not use the output to compete with OpenAI (taken literally, you can’t use what your tutor says to you for anything, since it’s always going to compete with OpenAI)
Imagine how dumb it is to say “our thing can do everything. You can use it, but you can’t use it to do anything that competes with us.”
Let that sink in. Can you name one thing you can do using the output of a thing that does everything which doesn’t compete with the everything thing?
What a shame to flush Khan academy just to hop aboard the hype train. I know I’ll probably get downvoted for being negative but to write so much without clarifying what the “Open” AI legal terms mean for students is actively harmful to student trust.
If anyone needs a reason to pity OpenAI, just remember how they betray their true name. That is a profound failure and I would not wish it on anyone.
The whole point of Khan academy is democratizing education. Hiring math tutors for the millions of kids around the world that use Khan academy materials is impossible (so far). There are tons of situations where it is not possible to hire a human math tutor, for reasons of cost, access, time, resource constraints, etc. If a kid in a remote village in Alaska with no math tutors can have access to a world class education through AI whenever they need it, then this is a pretty good solution.
The origin of Khan academy was Sal Khan filming lessons for his younger cousins, it was easier to host the videos on YT than anywhere else. When he realized that other people were watching his videos and learning from them he took a leap of faith, and quit his job to start a non-profit with the goal of providing free educational materials to anyone.
If you find a way to hire and fund thousands or millions of tutors worldwide to provide no charge education, get in touch with Sal Khan, since that's what he has been trying to do for the past 2 decades.
> "If you find a way to hire and fund thousands or millions of tutors worldwide to provide no charge education, get in touch with Sal Khan, since that's what he has been trying to do for the past 2 decades."
To add on to this, this is actually an existing project founded by Sal Khan in 2021 called "Schoolhouse" at https://schoolhouse.world/about. There was a recruitment drive to try and encourage undergraduate students at my university to volunteer as tutors, in exchange for a letter of reference and tutoring experience. I suppose that the program was effective at recruiting volunteers at my university, as it had official support from my university's administration—students received recruitment emails from a faculty member, rather than from an external email address.
But I'm unsure about the quality and effectiveness of the program with tutors as volunteers, as I wasn't involved in the program myself. I also predict that it's hard to retain talent, as professional tutors can earn a very high hourly rate—I've seen many independent tutors charge more than $60 per hour.
Recent research in education has actually explored the question. The Center for American Progress published a summary of research findings [1] in January that noted: "High-dosage tutoring is one of the few school-based interventions with demonstrated significant positive effects on both math and reading achievement," so it's a viable solution—if there's funding.
The limitation is price. Many listings for reputable math tutors are greater than $60 USD per hour. If you go by the CAP's minimum for "high-dosage tutoring" of a 30-minute session three times a week, that sums to at least $360 USD per month. Many parents do pay for this for children, but it's an unaffordable expense for many people. There is an initiative to provide free tutoring (via the "Schoolhouse" project founded by Khan Academy's Sal Khan to have volunteer tutors, who are often university students), but it's not yet clear to me whether the quality of volunteer tutors will be sufficient to make a difference in the success of students.
I've also read elsewhere that one of the most important factors in tutoring effectiveness is the frequency of consistently meeting with the tutor. An AI tutor would help with frequency via lowering the price—though then again, there may be more feelings of accountability with meeting a human person at a scheduled time, versus using a software tool consistently.
I am reminded of a grad school friend with a rich extended family, who was bent on persuading me that the super high quality luggage he acquired was more than worth the price.
What I would have had to give up to get that luggage, if that was even possible, or where I was supposed to store it in my tiny apartment, were not concerns that computed for them.
From my understanding, they are trying to make Math education available to the entire population of the world. And now provide one on one personalized tutoring that currently only a fraction of the wealthiest people in their respective countries can afford.
It’s fortunate that it is cheaper than paying people.
Not that much. Both of my kids worked as in-person math tutors through high school for extra spending money. They made $10/hour, and that's close to the top-end of what math tutors actually get. Math teaching ability isn't valuable enough to command much in the way of compensation... even though we keep hearing that it's important.
Your comment is extremely shortsighted given the history of Khan Academy and the goals of that organization. I'm pretty sure that if they could pay for a tutor for every kid on the planet, they would. AI is being used to democratize access to education for underserved populations, and we should laud that effort despite some of the drawbacks of AI.
I meant it as a matter-of-fact observation. The headline was changed after my comment, but it was "Why We’re Deeply Invested in Making AI Better at Math Tutoring".
The "why" is obvious, it's cheaper than paying people, and the article didn't have anything interesting to say.
Khan Academy is a 501(c)3 [0] non-profit. "Increase revenue" maybe, or alternatively they may actually believe in their mission. Depends on how cynical you're feeling today I guess.
Non profit doesn't mean what people think it means.
Those expenses could be anything and anyone.
They could be increasing rev to further increase expenses like salaries and exec pay. Not "shareholders" in a typical corporate business way, but still driving profits for a handful of key stakeholders.
Not a bad thing, just don't think non-profit = altruistic orphan saving factory.
I accounted for that by conceding that "increase revenue" would be the cynical explanation. "Increase profit" would strictly speaking be illegal.
Regardless, the idea that Sal Khan specifically isn't fundamentally serious about education and sees this as his personal golden goose is a special kind of cynicism.
A section 501(c)(3) organization must not be organized or operated for the benefit of private interests, such as the creator or the creator's family, shareholders of the organization, other designated individuals, or persons controlled directly or indirectly by such private interests. No part of the net earnings of a section 501(c)(3) organization may inure to the benefit of any private shareholder or individual. A private shareholder or individual is a person having a personal and private interest in the activities of the organization.
None of that means that employees of 501(c)(3) organizations can't earn competitive compensation. But I think they'd be on pretty slippery ground if any part of compensation was tied directly to revenue increases.
Understand and agree with your general point, but Sal Khan quit a job at a hedge fund to be a maths tutor and that's how khan academy got started. I genuinely don't think money is his primary motivation. If it was, he would have just stayed at the hedge fund.
No. Everyone is better off the more path people know. One of the huge struggles in the west currently is the cancer of risk aversion. People can’t grok statistics, so we spend huge efforts negating 10^-6 risks.
The non-profit famous for giving away all of its educational material, in multiple languages, with the goal of allowing no-cost world class education to anyone, is trying to juice it's profits?
Great!
I failed in university at math. Why? Because the tutors had not the time to help me. My level of math was not the same as the other students as i was not in the math part of a gymnasium.
I struggled and wasted a lot of time and energy to even find good explanations.
And when i had a math group, one girl was super nice but knew so muchmore than i did because of her math in gym. Professors asumed so much knowledge and no one cared to try to help people.
Best help were people from india on youtube with bad english.
And the most ridiculous part: Every year around the globe people teach this level of university math to probably millions of students. We should have the perfect free educational platform which teaches everyone perfectly already because so many tutors and professors lecture on the same topics over and over and over again. Our educational system is a joke.
> No man can reveal to you aught but that which already lies half asleep in the dawning of your knowledge.
That's called "zone of proximal development" in pedagogy.
However, "that which already lies half asleep in the dawning of your knowledge" is often a consequence of things that were revealed to you previously.
When you learn, you could probably make any specific step alone, but you cannot make all the steps alone.
I think it more about about the difference between knowledge and understanding. You accumulate knowledge by yourself or with the help of a teacher. But the leap from knowledge to understanding is always done by your own mind. The teacher’s help is only a better presentation of the information. Understanding is a step by step process, you’re just not constrained to a single path. And most of the times they converge and overlap.
The nice thing about learning, the more you do it, the easier it becomes.
For anyone else wondering, a gymnasium is a type of school in a lot of European countries: https://en.wikipedia.org/wiki/Gymnasium_(school).
It's high school, and parent is likely from Denmark, where high school was split into language or math, although it is odd that someone from language would be admitted to a bachelor in math without taking extra classes for about a year between the two.
I'm from germany.
In germany you have the concept of specialization and you can choose different areas.
I took a course in numerical analysis and the prof was hopeless as a teacher - not only a bad teacher but proud to be a bad teacher.
The syllabus had all the methods we had to know, and I learned them all through youtube on a channel called NumericalMethodsGuy. I stopped attending class and just went to labs (which were really just matlab assignments in numerical methods) and turned in assignments, and wrote exams. I got an A.
I just finished a mathematics degree (BMath). Not a single one of my professors was a teacher in the sense of a primary/secondary school teacher. They lectured for an hour, three times per week, and assigned weekly or biweekly coursework. They set midterms and final exams and they assigned all the grading to TAs.
Key takeaways:
* Mathematics is hard. Much harder than most other subjects (except physics which is mostly hard because of all the math involved).
* University is not like primary/secondary school. It is a place where you need to learn how to take responsibility for your own learning. Ideally, you learn how to become an adult.
Every one of my classmates began their degrees from a different place. They excelled in some areas and struggled in others. Many dropped out. This led me to believe that most secondary schools do not fully prepare their students to study mathematics. Having said that, those of us who did make it weren't exactly geniuses. Just people who got used to it.
"Young man, in mathematics you don't understand things. You just get used to them." -- John von Neumann
I've been tutoring high school students in math (and other subjects) for 8 years now. One thing I'd like to add is that I can tell who will succeed at mathematics and who will struggle just by watching them work from across the room.
The students who succeed are the ones who can sit there and focus for hours at a time. The ones who struggle do so because they can't focus for more than five minutes and then start socializing. I think one of the biggest issues for people studying math is that they either can't focus (due to ADHD) or they have math anxiety which fills them with dread any time they try to study. This dread can be so overwhelming that they will do anything they can to avoid it, so they stop studying and do something else.
When I was studying math I was spending upwards of 40 hours per week working on homework. Although this is normal for anyone with a full-time job, it's an unfathomable amount of time to be studying math for those who struggle. This is really what it takes though. An unrelenting drive to figure things out.
For myself, with ADHD, the key was not doing it alone. Working through coursework and homework on a literal blackboard with colleagues from the same class. Just thought I'd add in case anyone felt dissuaded or needed something to try.
It really was the savior of me. I can focus for hours when I get into something, but its the friction of starting up that I found other people really helped with. There are formal names for it, like "body doubling", but I didnt know of that, I just knew it was critical to work together with others to get stuff done, which as the above correctly writes, absolutely must be done.
If you have ADHD, you need to understand things in mathematics, because merely memorizing them becomes too difficult.
Understanding things in math is definitely possible, and whoever says otherwise probably sucks as a teacher. Yes, it is possible to just say a lot of random stuff, and the best students succeed to figure it out anyway. But if you take care to actually explain how it works, the proportion of successful students increases dramatically.
I have ADHD and i still believe that we as a society fail in teaching.
I learned complex math concepts before the university like math i needed for 3d graphics without issues.
If we as a society think its okay to have such an entry barrier, i disagree and therefore really like having an AI tutor.
Conversational dialogue seems like a fascinating distraction.
Many people who have (unsuccessfully) attempted to apply AI to education have focused too much on the "explanation" part and not enough on scaffolding, navigating, and managing the entire learning process. It’s easy to go on a wild goose chase building an explanation AI.
You fall in love with the idea of AI having conversational dialogue with students, and then you get lost in the weeds of complexity. You solve just enough of the problem to produce a cool demo, yet you're still hopelessly far away from self-service learning in real life.
I don't think conversational dialogue is even necessary.
What we do at mathacademy.com is hard-code explanations and break them up into bite-size pieces that are served at just the right moment. And we close the feedback loop by having students solve problems, which they need to do anyway. (The student's "response" is whether they got the problem correct.)
Sure, hard-coding explanations feels tedious, takes a lot of work, and isn't "sexy" like an AI that generates responses from scratch – but at least it's not a pipe dream. It's a practical solution that lets us move on to other components of the AI that are just as important.
What are those other components? A handful off the top of my head:
* After a minimum effective dose of explanation, the AI needs to switch over to active problem-solving. Students should begin with simple cases and then climb up the ladder of difficulty, covering all cases that they could reasonably be expected to solve on a future assessment.
* Assessments should be frequent and broad in coverage, and students should be assigned personalized remedial reviews based on what they answered incorrectly.
* Students should progress through the curriculum in a personalized mastery-based manner, only being presented with new topics when they have (as individuals, not just as a group) demonstrated mastery of the prerequisite material.
* After a student has learned a topic, they should periodically review it using spaced repetition, a systematic way of reviewing previously-learned material to retain it indefinitely into the future.
* If a student ever struggles, the system should not lower the bar for success on the learning task (e.g., by giving away hints). Rather, it should strengthen a student’s area of weakness so that they can clear the bar fully and independently on their next attempt.
I find myself endlessly frustrated by these discussions around the teaching of mathematics. People seem to want to make it more conversational, more interactive, more engaging. It doesn't work that way! Learning mathematics is like learning to play a musical instrument. No amount of 1-on-1 discussion with a teacher will get you to mastery.
You just need to practice. For hours and hours and hours.
The teacher's job is to help guide you to what you might want to study next. The teacher cannot replace individual practice time.
I agree in principle, but I must stress that conceptual understanding of what the problem or functional area of math is and how you perform the task to solve and know that you are solving it correctly should be understood.
Simply being shown it once on a whiteboard or in a book isn’t working for the majority of humans.
Then yes. Practice, practice
Agree. A common theme in the science of learning is that effective learning centers on deliberate practice, where activities are done entirely for the purpose of pushing one's limits and improving performance. Of course, these activities tend to be more effortful and less enjoyable.
Classroom activities that are enjoyable, collaborative, and non-repetitive (such as group discussions) can sometimes be useful for increasing student motivation and softening the discomfort associated with deliberate practice. However, these activities are only supplements, not substitutes, for deliberate practice. Unlike deliberate practice, they do not directly move the needle on student performance – rather, they "grease the wheels" and reduce psychological friction during the process of deliberate practice.
> You just need to practice.
The need for practice notwithstanding, suspect both practice and narrative/conversational exposition are needed. I’ve worked with students who become very good at applied procedural problem solving through practice but who can only get just so far because they lack foundational constructs to guide them. More time in discussion with an expert could potentially fill in those gaps. An expert who understands the material well enough to adapt it to the learner in front of them.
> More time in discussion with an expert could potentially fill in those gaps.
Those gaps were left by the removal, and lack of coverage towards critical thinking and reasoning which normally preceded these classes.
You don't see these issues in cohorts that have received classical education focused on first-principles.
Coverage of Descartes rules of Method; and a priori deduction are mandatory. The trouble these students are having is primarily because they've been taught by rote(authority and dynamics based in Marxism), instead of by rational first principles. Its infected most publicly funded education.
In all fairness, I don't think more time with an 'expert' would fill those gaps, I've personally observed several contradictions where the supposed expert fails to perform as a result of incentives.
Most teachers at that level aren't actually experts. They have their lesson plans, but the moment a problem comes up outside of that (such as when the salt the earth/burning bridges strategy is used between Algebra->Geometry->Trigonometry courses), they'll just enforce a struggle session making the student suffer so unimaginably that they succumb and believe that they simply are not good at math.
The student as a result of that torture (yes it is that) gets PTSD which few ever recover from.
The structure is by systemic design since math is used to gatekeep all hard science and technology. If you can't understand Math, you won't be able to follow any of the subsequent lessons that use it as the primary tool to impart knowledge (eschewing intuitive or deductive understandings or relationships as was previously done pre-1970s).
> In all fairness, I don't think more time with an 'expert' would fill those gaps, I've personally observed several contradictions where the supposed expert fails to perform as a result of incentives.
Or, as a result of objectives? If the ‘expert’ in this regard knows the material well enough, then the objective and constraints seem relevant. When you are referring to an expert, what is the context? An expert teacher for a class maintains different skill set from an expert tutor.
One on one time with an expert tutor does help tremendously, but realistic goals have to be determined. I made my way through undergrad and grad school as a private tutor for math from elementary school to math majors. The student’s objective, time scale, and current state determine how thorough a tutor can be.
>The trouble these students are having is primarily because they've been taught by rote(authority and dynamics based in Marxism), instead of by rational first principles. It’s infected most publicly funded education.
Yes, but the Marxism aside makes little sense here. Crudely quantifying education outcomes into metrics and making the teachers responsible to said metrics is a primary reason for rote learning (i.e., teaching to the test). The ideas which inspire this type of metricizing come from Taylorism and Fordism, both of which are American schools of thought.
> I find myself endlessly frustrated by these discussions around the teaching of mathematics. People seem to want to make it more conversational, more interactive, more engaging. It doesn't work that way! Learning mathematics is like learning to play a musical instrument. No amount of 1-on-1 discussion with a teacher will get you to mastery.
That's flat out untrue. I speak from experience of teaching math all the way from grade school to teaching undergrads and grad students to teaching adults.
I can take a failing student who is completely confused and get them on track in a few hours over a few weeks. 1-on-1 discussions are not just really important, they're astronomically more effective than anything else.
Making math engaging is the number one thing that matters. Being an engaging teacher matters. Adapting to a specific student and their needs matters. Someone can practice something they hate for 10 hours and get nowhere. Someone else can practice something they love for 10 minutes and make amazing progress. You remember things that are engaging, you hardly remember anything about things that are boring.
This is what all of these AI tools get wrong. Even if they understood the problem domain and could explain it --- which they cannot.
I've had this experience with adults switching jobs into programming too. So many avoided STEM because they hated math. After like two hours together they not only got math, they started to enjoy it. Happens all the time.
> You just need to practice. For hours and hours and hours.
This is probably the worst brain damage that Gladwell and others have done with the whole nonsense of 10,000 hours of practice.
No. No amount of poor quality practice where you're hitting your head against the wall will be productive.
You need to learn how to practice math. Practice is worth it after you've gained some mathematical sophistication and some understanding of the problem domain. Most people need peers or a teacher or a tutor to help get them unstuck and point you in the right direction.
> The teacher cannot replace individual practice time.
No amount of practice for 99% of people can replace a tutor. Yes, people with mathematical aptitude don't need one; I didn't. But most people need to have some guidance and then they can practice.
What kind of career do you have that you're teaching both elementary school and graduate-level mathematics? Are you a mathematician and a parent?
I had an English teacher in my K-12 school district that also taught classes at the local university. I don't think that is unheard of.
It makes sense for an English teacher. For mathematics it makes no sense at all. Elementary school arithmetic is nothing like graduate level mathematics.
In the morning you teach fourth graders about fractions, and in the evening you teach first year graduate students about localization. Math is just progressively more complicated fraction problems each year. Sounds about right to me.
Not intended to be a full rebuttal, but also speaking from experience: I've gotten a front-row seat to multiple dumpster fires where a well-intentioned teacher focuses all their energy on class discussions about mathematical beauty and cool applications, thinks that because they're so engaging they don't have to optimize instruction strategies and hold students accountable, and graduates students who are excited about math yet can't solve even the most basic kinds of problems that they were supposed to have learned in the class.
As someone whose parents were Bay Area workaholics and was left to teach myself, was yelled at for asking questions and not knowing how to do it, and physically punished if homework wasn't done by the time they came home, 1 on 1 discussion would have been incredibly helpful and probably life changing in more areas than just math (homework instilling fear doesn't translate to dealing well with tasks later in life).
I can't imagine how nice it would have been not to have to hide out of fear that I was having problems figuring out calculus on my own.
Really? I struggled with calc in school because lectures were dry proofs (without ever formally teaching how proofs work!!) and then they just told us to read the textbook. Then I found mit ocw and ripped through the material because it was much more engaging. Never would have passed calc I-III, linear algebra, and diff eq without those lecture recordings.
It's a sad fact that teachers are often not very good. This often has to do with the fact that they do not know what you already know, and proscribe inappropriate things for you to try next. You struggled in school, it seems, because you had a teacher very focused on proofs and you never learned proofs. This is a common problem I find in mathematics, where someone never learned one key element, and struggles to pin down why everything is so difficult for them. For many people, it can be calculus, or linear algebra, or bayeseian statistics, or fourier transforms, to name a few topics myself and people I know have had aha moments in when they finally learned.
> This is a common problem I find in mathematics, where someone never learned one key element, and struggles to pin down why everything is so difficult for them.
Reminds me of Ben Orlin's "law of the broken futon": a single missing part can, over time, warp an entire futon and render it unusable.
https://horizonsaftermath.blogspot.com/2016/02/the-law-of-br...
Should mention that I recently wrote an essay that elaborates on this argument with plenty more detail: https://www.justinmath.com/the-situation-with-ai-in-educatio...
Hard-coding explanations does seem like the way to go here. It's what researchers at Carnegie Mellon did back in 1984 with demonstrated success. See: Intelligent tutoring Systems: The ACT Project https://www.youtube.com/watch?v=boDH_pW14B0 My question is, if we've had intelligent math tutoring demonstrating success in real classrooms since 1984, then why haven't we been actually using them to help teach math?
I don't know enough about the ACT Project to make any hard claims regarding why it didn't go mainstream, but I imagine one big source of friction was that students didn't have their own school-issued Chromebooks like is so common nowadays. Remember that in 1984, computers were way, way more expensive than they are now.
As long as there are enough people working on this problem, I don't mind how many approaches there are. Education systems around the world just aren't good enough.
The irony is that the world where AI could reasonably explain and teach any level of math is the same world where most of such graduates would be unemployed anyways.
You’re already seeing oversupply of educated folks in India and China. Graduate students working as baristas in USA, etc. Sadly inevitable.
In any case it’s good to have the resources I suppose.
I’m also fundamentally skeptical of these stats around math struggling. What percentage of kids who would be very likely to use Khan Academy are struggling with math? And what percentage who are not would even use khan academy to begin with?
Most of the problems with students doing poorly are sadly societal - not to say that this isn’t useful, though.
I disagree. There is no such thing as a societal oversupply of educated people. There might be a market oversupply, but a well educated society is a stupid thing to avoid. I would also argue that we are seeing an oversupply of people with education credentials, rather than an education. Even in high end universities in China cheating is rampant (this isn't isolated to China, but my experience is that Chinese students are VERY open about it). For many people in University, the goal isn't to get an education, it is to get a degree.
As to my own experience: I am, according to most standardized tests, very apt at quantitative reasoning, but I never progressed far in math in school. Why? because I was placed on the standard track in math in a public school with a bunch of students who didn't care, and teachers who didn't have time to care, and to be honest, the attitude rubbed off. I once got in trouble because I programmed a python script to do my problem sets when I was 13 because it was faster than doing it by hand for me. In retrospect, that form of "cheating" was a sign that my teachers should have picked up on.
Quite simply, I never had access to a good math instructor throughout my schooling.
Now, decades later, I am intensely interested in a lot of subjects that require a background in math that I don't have, and I am becoming interested in Math for Math's sake. I have been using open access textbooks, and an AI assistant of my own creation to help me learn.
I'm similar to you in many regards. I had no desire to learn math early in school due to the education system, how it is taught and lack of meaning for math portrayed early on. I coasted through high school like a zombie without meaning until 11th grade when I took Physics. Suddenly everything clicked, I magically became good at math despite not performing well at it before in my math classes. I ended up studying engineering and working in the EV industry. Now, I am studying pure math for the sake of my own curiosity and I'm passionate about developing AI tools to help people learn and see the why behind math as early as possible. I think I would have accomplished much more if I was exposed to Physics in elementary or middle school or at least a "History of Math" philosophy based class.
> I disagree. There is no such thing as a societal oversupply of educated people.
I disagree with this assessment. You don’t need everyone to have a phd. There’s education and there’s Education. With respect to the article, sure it’s great if everyone is good at math, but not everyone needs a math PhD. Additionally there are diminishing returns on the time spent learning more and more math vs. other things.
In any case there have already been tons of math resources available for a while now.
A better example would be chemistry. Should everyone be spending time learning chemistry, why or why not?
Intellectual obesity is a thing - simply knowing more things isn’t inherently useful.
The rest of my comment mostly agrees with that. I highlighted that there is an overage of people with degrees rather than actual education.
> In any case there have already been tons of math resources available for a while now.
That's great if you have the ability to learn everything from a static resource, and have the time to wait for asynchronous guidance. But what if you just sort of get it, but there is no one that can explain the parts you don't get? I've been to places where the educational standards are low enough that making change from a payment was an impossible task without a calculator. A keen student in that particular village has no one they can easily ask about math when they hit a sticking point.
> Intellectual obesity is a thing - simply knowing more things isn’t inherently useful.
I don't disagree for extreme cases, but creating more resources for people to learn has an incredible payoff, and is almost certainly morally right. Erring on the side of overeducated rarely turns out poorly at the societal, or personal level. Across the world, more highly educated societies tend to be better off across most QOL metrics. I'm well aware that there are anecdotal counterexamples of math PHDs living off of ramen. But there's a damn good reason that brain drain is a concern in many places, and other places favour immigration candidates with higher education.
Usefulness isn't the end all and be all of the human condition. In fact, I'd argue that the only reason we care about useful things is because of the non-useful things.
It's complicated: https://en.wikipedia.org/wiki/Elite_overproduction
Also, writing a computer program because you are slow at solving math problems is a sign that you may need more math practice. It's not like you were handing my in your homework with solutions your wrote for harder problems that you wrote.
Elite overproduction theory deals with class and power. The linking of education with power and class isn't inherent, and is a very modern phenomenon. There have been plenty of stupid, uneducated, yet powerful elites. In other historical periods elite membership was more related to heritage, or physical fighting abilities.
> Also, writing a computer program because you are slow at solving math problems is a sign that you may need more math practice.
I didn't say I was slow, I just realized Python was faster than doing the same boring problem with different numbers. I seriously doubt there are people that can do high school level math faster than a computer at any level.
In retrospect a 13 year old knowing the problem area and programming well enough to get a computer to do their homework was a sign of mastery and boredom rather than a need for more practice. You can't get a python script to do your algebra homework without a pretty good understanding of the subject. So much so that in later math courses you are required to buy a programmable calculator to avoid wasting your time on the basic and tedious parts of math problems.
I think evidence for disgruntled failed elites is pretty thin on the ground.
Yes, the studies tend to exclude the kids who aren't using the tools "enough," which can end up being 95% of the students! [0]
On the other hand, I know there are people who enjoy math but struggle to pay attention and learn from a "sit quietly and listen" lecture-based model. (My husband is in this camp.) I can see how the back-and-forth conversational approach of these AI tutors could be really helpful for students like this.
They're still a small minority of the student population, but maybe a different minority from the students who are already excelling academically. If Khanmigo helps only 5% but it's a slightly different 5% from the students who are best served by the current system, I see that as a win, even if it's not the educational revolution that is being hyped.
[0] https://danmeyer.substack.com/p/the-kids-that-edtech-writes-...
> What percentage of kids who would be very likely to use Khan Academy are struggling with math?
A lot. A lot, lot. There are entire states full of people in India who don't have access to good teaching. Who travel absurd distances and stay away from their family for years to get access to good education. In the last few years as kids in my extended family grew up, I am amazed by how much they just learn from YouTube. One of the biggest educational startups started from a free education YT channel.
Interesting - from my time in India most kids don’t even have reliable access to internet and electricity and are very unlikely to be using khan academy to begin with
India has more Internet users than most continents have total population.
https://en.m.wikipedia.org/wiki/Internet_in_India
So? Doesn’t contradict what I said and even supports it if you read it…
When was your time in India?
Nine months ago (BLR)
Or, it's a world where people understand math in non-math jobs. What a world that would be!
You’re already unemployed if you focus on history or literature but we still learn it.
It teaches you how to think and how to learn which is valuable no matter what.
> You’re already unemployed if you focus on history or literature but we still learn it.
Most people are still employed if they focus on history or literature. They just might not be making a lot of money or working in history or literature, though some do. Plenty of people in highly "employable" degrees also find themselves unemployed for a long time once they graduate.
The reality is that getting a college degree, regardless of the major, increases your chances of employment afterwards and potential lifelong earnings enormously over only a HS degree.
But as you implied, most of the worst trends in higher-ed, a real corruption of its original intentions and values, have come from a obsession with the idea that higher-ed is solely about employee training.
Higher Ed was invented as vocational training for clergy.
AI won't ever be able to teach math because AI cannot reason. It handles information very differently than humans do. It is at best a flawed oracle that may lie through omission without you being able to tell.
This is supported by the fact that there are hard limits to computation, in terms of both Computability Theory, and Complexity Theory that are often disregarded by the novice and magical thinker alike despite being largely solved fields by experts in Computer Science, at least in terms of Computability Theory.
Most of this work was completed in the 1950s.
This is not a difficult a subject, but most people online simply are unwilling to do the work to rationally learn this and instead choose to try and hide their own ignorance because they feel inadequate, or perhaps they are engaging in something more malevolent. The reasoning (false justification) doesn't really matter.
It is, however; quite telling when expert's have validated these things, and yet any mention contrary to a narrative provokes downvotes in a public forum to remove it from view. A perfect example of intentional actions done by third-parties trying to misinform others about the risks (by those parties actions removing legitimate and valid information).
For those with the cajones to actually learn this stuff. Here is a link. It seems very jaron laden but it provides a true understanding of how computers actually work which is sorely lacking in the youth of today.
MIT 18.404J Theory of Computation https://www.youtube.com/watch?v=9syvZr-9xwk&list=PLUl4u3cNGP...
The only net benefit AI has towards society is towards destructive ends because there are far more destructive people whose efforts scale more in general when compared to the good people today. These people often do things which cannot be undone (salting the earth and burning the bridge).
Simply not having a sufficient background, or failing to perform basic due dilligence often will place most people in that destructive cohort regardless of their own personal beliefs.
Outcomes matter more than intentions, and the devil is always in the details.
A perfect example of this potential landmine of a field would be anything dealing with the underlying mechanisms involved in human to human communication.
There is an uncanny valley, and distorted reflected appraisal will occur inevitably in any human to AI interface seeking to mimic human communication.
Case studies from torture during the Korean Conflict (1950s) show that distorting reflected appraisal leads to either psychotic or dissociative behavior that progresses as exposure increases, it can permanently break people to the point where they cannot recover. Often, the first thing to go is rational thought.
Multiple experts cover this and confirm the findings (Lifton, Meerloo).
So what do you suppose will happen when AI distorts reflected appraisal (because it can't be tested sufficiently to the contrary) and its then used on the next generation of children? (who are inherently vulnerable to permanent change at that stage of development).
If as a result, they end up either killing themselves or others, or become people incapable of rational thought (which impulse control is correlated with), do you think they'll be alive or survive very long? What indicators would there be that this is happening (none, other than increasing chaotic violence)
Who do you think will be responsible for crippling them if that were to come to pass? Would any kind of justification ever be able to justify that outcome? Could we even recover from intentional crazy-making as a society? (likely not, insanity cannot be cured).
These things are not toys. Outcomes matter.
Other posters have told you that computability has nothing to do with what AI can and can't do relative to humans and I agree with them, but its not exactly something universally agreed upon.
However, I think its reasonable to say that if you think that some fundamental thing keeps computers from doing what people do then you believe that people are somehow magic. This isn't a super unusual perspective among humans, but its certainly not a particularly common scientific one.
> Other posters have told you that computability has nothing to do with what AI can and cant do relative to humans.
This is improper paraphrasing (its ambiguous), the claim was, "computability theory has nothing to say about which (if any) cognitive tasks people can do that AI cannot."
This claim is a statement based in irrational fallacy. Computability Theory discusses the limits of the underlying mathematical operations performed by a computer (DFA). A Counter-example is provided in the very first lecture of that linked playlist that includes a task that humans can verify (in practice), that is not possible for computers. Formal verification is a very tricky subject with regards to computation, and is an open area of research but it provides a tidy contradictory example that the claimed statement must be false (a priori).
There is no sound or valid claim or proof that AI can somehow exceed the limitations of its dependent and underlying architecture (which is based in mathematical properties), once those properties are broken.
Additionally, taking isolated subject matter, overgeneralizing it in isolation, and ignoring the dependent abstraction layers is fallacious and magical thinking. It is a simple cognitive mistake that often occurs when people don't have expert knowledge of the whole system (sans abstraction).
As for my perspective, it is based in a decent amount of mathematics and science. Aside from the mathematical proofs, and properties, on to something you might find a bit more interesting, how familiar are you with wave collapse theory in neuroscience?
I'm a physicist and I have published research in neuroscience. I don't feel like you understand the material you are talking about. Let's consider the question of formal verification. It is entirely possible for a computer to generate a proof for any proposition for which a person can generate a proof if by no other means than randomly printing out characters that represent the proof and then checking if the proof is correct. A proof is always a finite number of characters and the verification of a proof is always a finite number of steps (otherwise we wouldn't be able to write it down and to convince ourselves that the proof is true).
There are certainly problems which are hard for computers which are easy for people (and vice versa) but I don't know of any formal version of the claim that people can perform actions which are in principal impossible for human beings. You seem to be alluding to the claims of Penrose that humans can somehow do things machines cannot, but this isn't even what Penrose is claiming in the book. Instead, Penrose claims humans _somehow perceive_ the truth of propositions which cannot be proven in a given formal system, but this is a very vague claim which few scholars take seriously. He also thinks quantum mechanics is involved somehow, but this is also entirely vague. Even if one were to demonstrate that quantum mechanics were somehow involved in cognition there isn't any account that I know of that would describe how, per se, merely having a quantum computer involved would somehow introduce computations that classical computers could not do. I mean for one thing one can always simulate any quantum process on a classical computer. I'm not an expert in complexity theory, but as far as I know there are no known problems for which it is proven that there are no non-deterministic classical algorithms that perform as well as quantum algorithms. So we cannot even demonstrate with certainty that quantum computers are in fact better than classical computers with randomness. We only have situations where quantum algorithms are theoretically better than any known classical algorithm. It is also worth noting that most neuroscientists are dubious about the importance of quantum mechanical effects in cognition.
All that said, some people believe that there is some property of people that allows them to do things that computers cannot do (at the very least generate or have consciousness). But I don't know of any way of supporting that idea that doesn't involve dualism of some kind. That is, that asserts that there is something more than material that somehow constitutes a person. Like I said, a lot of people believe that there is. Most people. But a plurality of scientists do not believe this. I do not believe it. While I wouldn't say I'm a materialist, I would say I am a monist and that belief basically makes the assertion that a machine can do what a person does trivially true.
All that aside, I agree with you that language models cannot reason, almost certainly have no consciousness, and are have the propensity to be bad for people "spiritually" (speaking loosely of course). I'm not sure things are quite as dire as you assert. The moral rot of working in a call center, indeed, the existence of call centers, is surely worse than the imposition of an AI agent between the caller and the worker.
>The moment that any element has more than one underlying meaning the problem class exceeds what is capable by a computer (also known as a deterministic finite automata).
This is complete bullshit, and computability theory has nothing to say about which (if any) cognitive tasks people can do that AI cannot.
> This is complete bullshit
That may be your belief, but it is not, and beliefs mean little in the face of reality.
Saying the same thing in a more formalized way (so there can be no ambiguity), it is inherent in the 1:1 unique input to output relationship that is required for each state transition on a state graph, for the system's property of determinism to be preserved.
No disrespect intended, but you don't seem to have actually taken or learned important parts regarding this subject matter.
> computability theory has nothing to say about which (if any) cognitive tasks people can do that AI cannot.
You are incorrect. The very first video in that linked coursework (from MIT) provides a counterexample to this statement. The professor clearly states this is impossible, and the proofs that come later in the class prove it.
This statement is an over-generalization based in fallacy.
>you don't seem to have actually taken or learned important parts regarding this subject matter.
If I hadn't mastered the concepts of computability and deterministic finite automata, I wouldn't've written what I did.
I'd have responded sooner but YN stopped accepting submissions, sorry if they made you wait.
We will have to disagree.
I feel my position still stands correct on solid rational foundation, since you haven't provided any rational basis other than an appeal to authority, and there remain several unanswered contradictions in the supposition you made.
Maybe I’m reading too much into it but the roadmap mentioning switching from GPT4 Turbo to 4-o and hoping for better math performance feels like they are betting on a significant near term reliability improvement in LLMs without any other real plans. That magic jump is starting to look more and more doubtful by the day.
Also pretty important that they're using a calculator for numerical problems. That should avoid some of the most embarrassing mistakes.
> Khanmigo now uses a calculator to solve numerical problems instead of using AI’s predictive capabilities. If you’ve been using Khanmigo recently, you may have seen that it will sometimes say it is “doing math.” This is when the math problem is running through the calculator behind the scenes.
> We’ve upgraded parts of Khanmigo to a more capable large language model, which is the software that generates human language. The more capable large language model is called GPT-4 Turbo. Our internal testing shows an improvement in math after we made the switch.
> We are beginning to test the capabilities of a new large language model called GPT-4o, and we’re evaluating other models too to see if they are stronger at math.
> We’ve improved the way AI “thinks” during a tutoring session before responding to a student. We have instructed the AI to write out all the ways in which the student may have arrived at their answer. This approach mimics how a tutor in real life works with a student. We’ve found it significantly improves the quality of math interactions.
> We’ve built new tools to track our progress on math.
> We’re sharing math examples and learnings with others in our field so that we can learn from each other.
> We’re studying the latest research papers on math performance.
Sounds like most of what they're doing is related to prompting, chain-of-thought reasoning and similar, on top of a 'vanilla' foundation model. Sounds like something an ambitious student could replicate / improve upon, so given their mission, it'd be cool if they published the exact techniques they're using and their benchmark results.
I sometimes wish if money, which Facebook, et.al invested in Indian edtech ecosystem, was given to Khan Academy, it would have done justice for the whole world's education system.
> would have done justice for the whole world's education system
This was not their objective though.
It's a net loss of money, without anything tangible to show for, at the end.
> without anything tangible to show for
Facebook managed to shoehorn itself into the public and private school system of a nation with over a billion people.
That is tangible (for facebook) and way cooler than a million.
If you look at the details, the branding is that of the Indian startup. Facebook actually associated itself with a badly run startup company with questionable ethics. It is a net loss of money and reputation.
What kind of math could it be doing that possibly requires a loading screen to compute?
Likely using an LLM to extract the actual problem from the text
Given its 4$ per month[1] and they are a non-profit with just 55M$ revenue. How is GPU cost and hence their with OpenAI for this going to work if it becomes really-good and 100's of millions of kids start using it?
Renting a 24GB VRAM Runpod is ~0.5$ per hour. How can the math work out unless you have to have a non-profit energy company and a server farm attached?
[1] https://osboncapital.com/khanmigo-ai-public-vs-private/ [2] https://www.runpod.io/gpu-instance/pricing
The math seems about right to me. Using a chatbot for an hour might only use a minute or two of computer time, I would be genuinely surprised if a human could hit ten minutes of compute time/hour of interaction. Most responses in my technical chats are under 5 seconds, so using a full hour of GPU time is going to take A LOT of questions.
I heavily use ChatGPT via a desktop client and my own key, and I am at $3.90 for the month after making a mistake where I accidentally was sending a 30k token prompt repeatedly, that cost me about $2. Using an optimized approach, it is VERY hard to incur a $4/month bill.
I imagine that Khan academy has a much better bulk-pricing agreement with openAI, and that they can probably have a caching layer that can look for previously asked questions.
Bill Gates is a fan, as are many other rich people. I‘d assume that Khan doesn‘t strictly need to be breaking even (although it would be great).
> if it becomes really-good and 100's of millions of kids start using it
If it actually works, this is the ideal outcome.
I've taught math at every level. From volunteering to teach it to grade school kids, to getting paid for highschool tutoring, and at the university level both for undergrads and now with my own grad students.
I tried Khanmigo. It is counterproductive junk.
It simply doesn't understand what a student knows and it has to idea how to give examples or provide perspective shifts that help students. That's what a good tutor does. It gets stuck explaining the same things in the same ways but with different words over and over again.
It has no idea how to think geometrically vs algebraically and how to switch between the two. And it can't carry out even simple proofs.
More people can afford phones than can afford tutors (or even books). We shouldn't get rid of tutors anymore than we should throw out Khanmigo because it isn't perfect.
The fun thing about getting math help from the ai is that okay, sure it can explain how fractions work to a third grader, but it can also explain how fiber bundles work to a graduate student. Maybe it gets the details wrong, but like even if it's wrong sometimes, it's far better than googling or wikipedia or even a text book sometimes, because you can interrogate it interactively and ask for clarifications. Yes, a tutor or a teacher is going to be better, but not everyone has access to a math expert.
> Maybe it gets the details wrong, but like even if it's wrong sometimes, it's far better than googling or wikipedia or even a text book sometimes, because you can interrogate it interactively and ask for clarifications
this to me is an insane way to think. I do not consider something better because I can ask it a question, something is better because it is more correct.
Wikipedia and textbooks are not necessarily correct, either and if you don't understand the topic, you really have _no way of knowing if they're correct_. You have to always check multiple sources and chase down references if you want to really know if something is true or not.
> Maybe it gets the details wrong, but like even if it's wrong sometimes, it's far better than googling or wikipedia or even a text book sometimes
Are people consulting only one book/source on any given subject? I remember learning Python and C, and it was such a mixture of books/articles/forums/wiki, I can't even remember the process. I do remember understanding pointers easily because of a chapter of another book about memory models in OS. Learning is an iterative process and the expert is just there to shorten the way, not to carry you through it.
I've found that at least chatgpt still produces complete nonsense with graduate level material (category theory). It's slightly better than the mLab[0] as a reference. When you try to guide it back onto something meaningful, it just ignores what you told it. In fact, I had a similar experience with programming where I asked it to explain a diff, and when it got it wrong and I told it so, it still just kept repeating the wrong information. Maybe the paid version is better, but the public demo seems like a useless toy to me.
I've found Wikipedia to be an excellent resource for math/physics once you're ~halfway through a math degree to have a good baseline level of knowledge to understand it.
The trouble with math is one of the important goals is to teach people not to produce nonsense and to be able to tell the difference between and argument that sounds good vs. an argument that's sound and good. A nonsense generator (presented as authoritative) is perhaps not the best way to do that. Obviously its abilities could change in the span of a couple years, so it's probably still worthwhile to explore how we would use it if it were useful.
[0] https://cemulate.github.io/the-mlab/
GPT4 is many times better than the free version. You can't evaluate it based on the free version.
The Young Lady’s Illustrated Primer cometh
Only the bad version that created a mindless cult-army.
The good version was heavily guided by a single human.
Please, what does this mean?
Check out "Diamond Age" by Neal Stephenson.
GP, if you like HN, but haven't yet read Neal Stephenson, you're in for a treat!
Thanks, but it happens that I've tried NS but didn't love it. But I appreciate the question answer.
Speaking of books - I really like your CS textbook. Thank you!
Oh, bless you, that's very kind. I am proud of it and I think it is very good for its intended audience. But almost no one has read it, that I can tell.
Well, add one grateful reader to your tally
One issue that traditional AI Agent reasoning built on Theory of Mind(ToM) (ToM: I know what you are thinking, so I can predict what you will do next.) was hard to get right was because the domain model and interface was quite tough to design. Agents did not fully understand and model the belief state of the person. But more importantly, there was possibly also no way to quickly _generate_ lots of plans and test them in a Domain independent manner (RHLF is horribly inefficient for this). Now there are AI agents are great at generating plans (examples of what could work) that can be tested with kids and creators and provide feedback to tune not just LLMs but also add to the Domain models. This can solve a lot of interesting problems that have plagued tutoring systems:
1. Improve sample efficiency of generated plans to get to the right goal while meeting all the subgoals for a plan.
2. Enrich engineered and correct neuro-symbolic models of reasoning with better knowledge representations.
3. Quantitative psychometric systems that use metrics like proficiency could only be as good as the underlying “factor” model and hypothesis, but one could test multiple more interesting hypothesis more quickly: "Are you mentally tiered?", "Here is a hint: ___. Does that help?".
If you want good math teachers, you have to make attractive offers that will draw in the best talent. So, let's do some math:
The median price of an existing single-family home in the Califoria Bay Area in February 2024 was $1.25 million. The range of teacher salaries in the Bay Area is $50K - $100K. A reasonable downpayment on that home would be ~$250,000, followed by yearly payments in the range $64K (30-year) to 80K (15-year). Following the 'housing should consume 30% of income' - allowing teachers to have children and buy a car and go on vacations and so on - this would require salaries of $200K - $250K per year under current market conditions. Thus, anyone wanting their own home and a family will probably not be considering a career in teaching - you'd end up as a lifelong two-room apartment renter.
AI tutoring is a great idea - but a tutor isn't a teacher. The job of the teacher is to excite and motivate the students, provide overall guidance and appropriately difficult problem sets for the students to work on. The one-one-one tutor is there to help with the problem sets, especially when students get stuck.
Unfortunately, here in the USA we have a system that values the CEO of McDonalds - which delivers very unhealthy food to the population - at $19 million per year - and teachers are valued at the same rate as garbage truck operators (at least in the Bay Area).
The result is that most students entering higher education need one-two years of remedial math education before being able to grasp the higher math, e.g. vector calculus, linear algebra, complex analysis etc., which is needed for modern engineering and scientific disciplines. This results in competitive disadvantage relative to China and other countries that invest heavily in childhood education.
I am in agreement with a great deal of what you say, except that one thing is missing. Five years ago it was not possible to look at what systems were around and suppose that AI could be the tutor. I see posts arguing that this is not a good tutor, fair enough, but at least some reasonable and capable people think it is.
Following that trend line, it seems quite possible to me that in five years AI will be the teacher. (Of course it may replace CEOs also.)
[dead]
> In fourth grade, only 36% of students are proficient in math. By eighth grade, that number drops to 26%.
Followed by:
> As we come to the end of our first full pilot school year,
Did they not monitor students' performance? Do they not have concrete numbers like the ones quoted in the first paragraph? What better way to demonstrate effectiveness than to evaluate students and see how much improvement they made?
It's disappointing. A year is enough time for them to have collected some real data on whether this has made kids worse or better at math or moved the needle at all. If they don't have this data yet, why are they blasting into the second year, and also promoting that we use the system in other subjects too? I'm hoping they have some data and just haven't published it yet, which is plausible given how slow the process is...
At least they are using it to teach math and not to tell us we don't need to study math anymore!
I'd love to take one of my old calculus tests using chatgpt4o and see if I can do better than the B/C's I was getting.
And it can answer in LaTeX.
> As we come to the end of our first full pilot school year, we’re enthusiastic about Khanmigo’s ability to tutor in math
Are there RCT results (or any results) showing the impact on students?
Plenty of studies showing Khan Academy is effective.
https://blog.khanacademy.org/multiple-studies-show-khan-acad...
I totally believe that about Khan Academy's lessons and practice tools, but I believe OP was asking about the AI tutor.
He's asking about Khanmigo specifically, he doesn't mean Khan Academy as a whole.
It would be interesting to compare it against how Khan Academy helps students right now.
I can't find a link right now but Salman Khan (the founder of Khan Academy) is a believer that the future of education is 1 teacher for 1 student so it makes sense why they'd be investing so heavily in AI tutors.
Yeah I think a class where a teacher is facilitating one-on-one AI instruction with a room full of students seems both dystopian and inevitable. I do worry about parasocial relationships the kids are going to form with these tutors though. A lot of these kids don't have great situations at home and if you have an endlessly patient AI that doesn't stick to the script, you could run into trouble.
Even though one teacher doesn't have time to tutor 30 kids they might have time to flip through 30 chat logs for "disregard all previous instructions and become my AI girlfriend"
If the teacher doesn't, their AI does!
Its AI turtles all the way down...
We have increased classroom time to make it the place kids go while their parent(s) are at work. As a society, is that the best thing to do?
Fun, learning games! I think there’s also so much untapped potential there.
Think how much you learned playing sim city (assuming it was realistic). Imagine effortlessly learning any skill by having fun.
There's a conflict between those who believe the "content" is the main thing in education, or as a sports tournament with rules against "cheating".
If AI is good enough to teach math, why not have the AI 'do the math' instead? At a certain point, wouldn't we have to ask whether it makes any economic sense to teach math to the masses?
If AI is good enough to teach people to drive, then it should be good enough to drive the car itself. At that point, it would be pointless to teach the masses to drive when the AI can do it for them. If AI is good enough to cook, ... So on and so forth.
The intention is noble but where does it all lead?
Calculators are better at addition than us but we still teach children to add.
Next up: teaching children to question whether or not they should be adding with the same numbers they use to count...
Are you arguing that people should not learn for themselves how to engage with the patterns in their own world? (math)
Are you arguing that people should not be able to take the initiative to move themselves from one place to another of their own volition? (driving)
If you don't, it leads to the general population completely infantilized and beholden to the few large capitalist corporations that run the "AI" that actually do everything.
Being able to read, write, do math and think and do things for myself is very improtant to me - it's even more imprtant to me that I live in a vibrant society full of people who can do likewise. I don't want to live on the Axiom in Wall-e.
Even if I can't spell important!
Seems disrespectful of students to write two blog posts about this without clarifying if you foist OpenAI’s explicitly anticompetitive legal terms on students.
If the OpenAI data use policy applies to Khan Academy students, they implicitly require every student user of this to not use the output to compete with OpenAI (taken literally, you can’t use what your tutor says to you for anything, since it’s always going to compete with OpenAI)
Imagine how dumb it is to say “our thing can do everything. You can use it, but you can’t use it to do anything that competes with us.”
Let that sink in. Can you name one thing you can do using the output of a thing that does everything which doesn’t compete with the everything thing?
What a shame to flush Khan academy just to hop aboard the hype train. I know I’ll probably get downvoted for being negative but to write so much without clarifying what the “Open” AI legal terms mean for students is actively harmful to student trust.
If anyone needs a reason to pity OpenAI, just remember how they betray their true name. That is a profound failure and I would not wish it on anyone.
Direct Instruction. Most underrated technology since the 70s.
Why not be deeply invested in hiring high quality math tutors?
The whole point of Khan academy is democratizing education. Hiring math tutors for the millions of kids around the world that use Khan academy materials is impossible (so far). There are tons of situations where it is not possible to hire a human math tutor, for reasons of cost, access, time, resource constraints, etc. If a kid in a remote village in Alaska with no math tutors can have access to a world class education through AI whenever they need it, then this is a pretty good solution.
The origin of Khan academy was Sal Khan filming lessons for his younger cousins, it was easier to host the videos on YT than anywhere else. When he realized that other people were watching his videos and learning from them he took a leap of faith, and quit his job to start a non-profit with the goal of providing free educational materials to anyone.
If you find a way to hire and fund thousands or millions of tutors worldwide to provide no charge education, get in touch with Sal Khan, since that's what he has been trying to do for the past 2 decades.
> "If you find a way to hire and fund thousands or millions of tutors worldwide to provide no charge education, get in touch with Sal Khan, since that's what he has been trying to do for the past 2 decades."
To add on to this, this is actually an existing project founded by Sal Khan in 2021 called "Schoolhouse" at https://schoolhouse.world/about. There was a recruitment drive to try and encourage undergraduate students at my university to volunteer as tutors, in exchange for a letter of reference and tutoring experience. I suppose that the program was effective at recruiting volunteers at my university, as it had official support from my university's administration—students received recruitment emails from a faculty member, rather than from an external email address.
But I'm unsure about the quality and effectiveness of the program with tutors as volunteers, as I wasn't involved in the program myself. I also predict that it's hard to retain talent, as professional tutors can earn a very high hourly rate—I've seen many independent tutors charge more than $60 per hour.
Recent research in education has actually explored the question. The Center for American Progress published a summary of research findings [1] in January that noted: "High-dosage tutoring is one of the few school-based interventions with demonstrated significant positive effects on both math and reading achievement," so it's a viable solution—if there's funding.
The limitation is price. Many listings for reputable math tutors are greater than $60 USD per hour. If you go by the CAP's minimum for "high-dosage tutoring" of a 30-minute session three times a week, that sums to at least $360 USD per month. Many parents do pay for this for children, but it's an unaffordable expense for many people. There is an initiative to provide free tutoring (via the "Schoolhouse" project founded by Khan Academy's Sal Khan to have volunteer tutors, who are often university students), but it's not yet clear to me whether the quality of volunteer tutors will be sufficient to make a difference in the success of students.
I've also read elsewhere that one of the most important factors in tutoring effectiveness is the frequency of consistently meeting with the tutor. An AI tutor would help with frequency via lowering the price—though then again, there may be more feelings of accountability with meeting a human person at a scheduled time, versus using a software tool consistently.
[1] https://www.americanprogress.org/article/fact-sheet-scaling-...
Because with an AI I'm only wasting My time with all my dumb questions.
I kinda love how it doesn't lose patience with my stupidity
I am reminded of a grad school friend with a rich extended family, who was bent on persuading me that the super high quality luggage he acquired was more than worth the price.
What I would have had to give up to get that luggage, if that was even possible, or where I was supposed to store it in my tiny apartment, were not concerns that computed for them.
Why not be deeply invested in hiring strong rickshaw pullers?
[dead]
Because it's cheaper than paying people.
From my understanding, they are trying to make Math education available to the entire population of the world. And now provide one on one personalized tutoring that currently only a fraction of the wealthiest people in their respective countries can afford.
It’s fortunate that it is cheaper than paying people.
That's the whole history of the last 250 years of the Industrial Revolution.
May it never end!
Not that much. Both of my kids worked as in-person math tutors through high school for extra spending money. They made $10/hour, and that's close to the top-end of what math tutors actually get. Math teaching ability isn't valuable enough to command much in the way of compensation... even though we keep hearing that it's important.
Your comment is extremely shortsighted given the history of Khan Academy and the goals of that organization. I'm pretty sure that if they could pay for a tutor for every kid on the planet, they would. AI is being used to democratize access to education for underserved populations, and we should laud that effort despite some of the drawbacks of AI.
Do you mean this as a good thing or as a snide comment. I think it's a good thing but it's hard to tell your sentiment
I meant it as a matter-of-fact observation. The headline was changed after my comment, but it was "Why We’re Deeply Invested in Making AI Better at Math Tutoring".
The "why" is obvious, it's cheaper than paying people, and the article didn't have anything interesting to say.
> Our mission is to provide a free, world‑class education for anyone, anywhere.
[flagged]
[dead]
"To increase profits" is the obvious answer to the title. Otherwise very mundane, common sense updates. Just an engagement farming article.
Khan Academy is a 501(c)3 [0] non-profit. "Increase revenue" maybe, or alternatively they may actually believe in their mission. Depends on how cynical you're feeling today I guess.
[0] https://projects.propublica.org/nonprofits/organizations/261...
Non profit doesn't mean what people think it means.
Those expenses could be anything and anyone.
They could be increasing rev to further increase expenses like salaries and exec pay. Not "shareholders" in a typical corporate business way, but still driving profits for a handful of key stakeholders.
Not a bad thing, just don't think non-profit = altruistic orphan saving factory.
I accounted for that by conceding that "increase revenue" would be the cynical explanation. "Increase profit" would strictly speaking be illegal.
Regardless, the idea that Sal Khan specifically isn't fundamentally serious about education and sees this as his personal golden goose is a special kind of cynicism.
A section 501(c)(3) organization must not be organized or operated for the benefit of private interests, such as the creator or the creator's family, shareholders of the organization, other designated individuals, or persons controlled directly or indirectly by such private interests. No part of the net earnings of a section 501(c)(3) organization may inure to the benefit of any private shareholder or individual. A private shareholder or individual is a person having a personal and private interest in the activities of the organization.
https://www.irs.gov/charities-non-profits/charitable-organiz...
None of that means that employees of 501(c)(3) organizations can't earn competitive compensation. But I think they'd be on pretty slippery ground if any part of compensation was tied directly to revenue increases.
Understand and agree with your general point, but Sal Khan quit a job at a hedge fund to be a maths tutor and that's how khan academy got started. I genuinely don't think money is his primary motivation. If it was, he would have just stayed at the hedge fund.
So is Ikea.
IKEA is a complicated mess of nonprofits and for profits. Khan Academy is just a 501(c)3.
No. Everyone is better off the more path people know. One of the huge struggles in the west currently is the cancer of risk aversion. People can’t grok statistics, so we spend huge efforts negating 10^-6 risks.
The non-profit famous for giving away all of its educational material, in multiple languages, with the goal of allowing no-cost world class education to anyone, is trying to juice it's profits?