Rendered at 22:34:14 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
khalic 10 hours ago [-]
The tech is very cool but for the love of Gaia stop calling it system one, even Kahneman said this (S1/2) is a framework for understanding the brains inner workings. There is no autonomous system to speak of.
robrenaud 6 hours ago [-]
Brains aside, it S1/S2 seems like a good description?
System 1 is quick and intuitive. System 2 is slow and deliberate.
Compared to models that compose responses with many forward passes, thinking in token space, these models make a quick decision after processing their inputs.
khalic 5 hours ago [-]
It’s ok to use it as inspiration, but using the term comes with psychological baggage in an industry that already abuses anthropomorphism. Give it an informative name
aoeusnth1 5 hours ago [-]
System 1 is quite informative about what it's for. It means it's a system designed for bounded time quick classification, vs. agentic unbounded or slow thinking with scratchpads and tools.
khalic 4 hours ago [-]
Does it inject thoughts into system 2 without system 2 knowing about it? Does it have instinctive social skills? Can it do instinctive arithmetic? The list goes on, there’s a nice book about it…
Zarathruster 9 hours ago [-]
This is gonna be the new "I'm more of a left-brained person" thing isn't it
bbor 5 hours ago [-]
Ok sorry for the unprompted rant but you're absolutely right, though it's far from "new" and not really a problem (IMHO). AI even beat neuroscience to this one, funnily enough! All of the below dyads are congruent:
--- Phil & [Pre-]CogSci ---
| rational | intuitive |
| reason | understanding | (Kant & Hegel's goofy wording)
| deductive | inductive |
| animated | automatic | (from ~Aristotle, ultimately)
| ~higher | ~lower | (as in "higher faculties")
--- CS & AI ---
| logical | analogical |
| symbolic | stochastic |
| Neats | Scruffies | (related academic 'camps')
| ~deterministic | ~non-determnistic | (a common heuristic)
| Good Ol' Fashioned AI (GOFAI) | Evil Datacenter AI (EDAI) | (a common honorific)
--- Modern [Cog-]Neuro[-Psych] ---
| slow thinking | fast thinking |
| S2 | S1 |
...ok I assumed I knew more but maybe I don't? Would love to hear from experts!
--- Colloquial ---
| left brain | right brain |
| intelligence | wisdom |
| intention | instinct |
| me | AD[H]D |
At least that's how I see it. Hopefully it goes without saying that there's plenty of nuance specific to each of those pairs, and that many of the relevant experts would balk at this broad characterization. That said... I'm right and they're wrong, I guess!
If you prefer geniuses that aren't arguably monsters, this is the core of what Gary Marcus' "neurosymbolic" thing is about.
khalic 9 hours ago [-]
moan
rgbrgb 6 hours ago [-]
what's wrong w "system one"? it feels like a good frame to think about use-cases that will fit jev etc.
vatsachak 5 hours ago [-]
Why is ML so misleading these days? No this model does not score 80%+ on DeepSWE, it merely chooses the best possible idea of Opus 5 at every stage, increasing the performance by 5%.
aoeusnth1 5 hours ago [-]
Yeah, the github landing page is much less misleading and clearly says this. I don't know why they try to make the marketing page gloss over this detail... it speaks to the mindset of the authors.
mugul 16 hours ago [-]
Very interesting insight on the training process, it's pretty cool to have some experimental justification for why they took these exact steps, what they tried and did not work, etc. Feels a bit less like dark magic.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
amluto 13 hours ago [-]
I’m fascinated by this thing and by the way it’s interpreting Jev. It’s very cool, but is it actually a classifier?
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
visarga 7 hours ago [-]
> The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays
This is self contradictory. You can tell a correct answer as you said, by looking at the score. In my $DAYJOB I am making hundreds of RL environments that produce a score for each intermediate state.
drob518 6 hours ago [-]
Right, and I’d add that in some games there is actually a “win” state that serves as a binary indicator. Given this, you can actually view the model as the thing being classified into two states: (1) consistently wins the game and (2) doesn’t consistently win the game.
noduerme 13 hours ago [-]
I'm increasingly of the opinion that stuff like Jev and this are overfitting and producing illusory but confident "probabilities" that are complete bullshit. You can't see the underlying reasoning... but it's incredibly tempting for people who want to place faith in them. Rather than even trying to understand the complex system at play, it's easy to give up on trying to find reason, and just accept the second- or third-tier outputs of massively complex things that, on the first tier, are not necessarily reliable sources of truth.
Let's give an example: Suppose you ran a Jev that tried to determine when another Jev was wrong about something ...baseball games, let's say. The second Jev would come up with a perfect list of when the first Jev was wrong and when not to believe it. So now you have a second-order system that you believe more than the first-order one.
Only after 100 baseball games, the second-order Jev is only as good as the first one, it just inverted a bunch of games that could have gone either way. So hell, you make a third-order Jev that analyzes the first two...and its results are AMAZING when you look at the historical record! Only, you know, that's what's called overfitting.
Honestly, [edit: Fuck, I just wrote "honestly". I've been brain damaged by you-know-who] the invention of the "noul" is a bit of a giveaway. Imagine what junk bond traders could've done with that in the '80s. Not-not-not-falsy is how we all like our stock picks served up, right?
Reading the raw statistical output of an LLM as if it were an oracular source of truth is literally idol worship and gambling in the purest sense.
Take it from a guy who lost $10k on baseball this season having Claude rewrite my original code to reverse and re-reverse underdogs and favorites; there is no fucking "probability" of anything coming out of an LLM, even if your source of truth isn't an LLM but an evolutionary algorithm you designed yourself. Which would've performed better before the LLM started interpreting ways to bucket it and make up bullshit probabilities around it.
Anyone going down the Jev path is deeply misguided, but will see the light once they realize they have re-invented the magic 8 ball. Or that executive decision maker cube from the 80s with 6 random answers.
Here's your Jev. Note the preponderance of nouls ;)
If you happen to gamble or just watch baseball, you'll see that reality is not so easily reduced to one number. On the most basic level, a single probability number leaves out volatility, without which it's utterly useless for predicting anything other than a sequence it's already trained and fitted for. But as a gambling addict and occasional patron of oracles and gurus, it's very easy and tempting to mistake a clear-cut "probably" for whatever you want to hear. Listening to an LLM's heartbeat for probabilities is just a gambler's fallacy taken out to the 19th degree.
visarga 7 hours ago [-]
Yes, Jev is calibrated "from factory" on a bunch of tasks, but we can be almost sure our own bespoke tasks are not covered. So the model does not really know how to produce calibrated confidence scores.
What it would need is a calibration dataset on which to align. There is no calibration in the abstract, only relative to a set of test examples. A model with an uncalibrated output probability can be recalibrated using conformal prediction. You run the model over your calibration examples, get the probabilities.
Assume the new example's answer is y, and calculate its nonconformity score, higher means a worse fit. Count how many calibration examples have a score at least as high as that. Add one to this count, then divide by the total number of calibration examples plus one.
htk 10 hours ago [-]
I love this answer and your fondness for "assisted" betting. Do you blog?
noduerme 9 hours ago [-]
Thank you for that. But I don't.
I can tell you though, I'm not alone... a completely degenerate cokehead at my local bar just showed me a baseball prediction app he had Claude build for him on his phone, which looked suspiciously similar to mine. (Mine might be hand coded and backed by a symbolic regression A-Life engine I've been working on since 2005, but so what? Claude has helped me flip most of the predictions for various reasons, to equal "success"). Crude, and yet his app somehow included wind direction forecasts for each inning of every game. This guy is smart as a punter but has probably never used a computer for anything in his life beyond downloading STD test results.
It's a brave new world.
fxwin 15 hours ago [-]
I really hope that "System One" won't stick around as a new buzzword simply meaning "fast".
i'm well aware of the origin and meaning of the term
> "System 1" is fast, instinctive and emotional
this implies more than just "fast", which is precisely why i don't like its present usage
tancop 14 hours ago [-]
Instinctive is a good way to describe it compared to generative LLMs. Jev gives you one instant answer, fast and usually correct but without nuance or any explanation. Human instincts work the same way.
fxwin 12 hours ago [-]
You're describing external aspects but to me, "instinct" says much more about internal processes than the properties you mentioned, and I haven't seen anything that tells me how these models draw on anything similar to these internal processes to generate their outputs (at least not more than generic LLMs do)
__alexs 13 hours ago [-]
In my testing Jev is not what I would call "usually correct" on most topics that involve knowledge of the world outside of the context you give it.
TeMPOraL 9 hours ago [-]
Which is why "System 1" fits even more.
__alexs 7 hours ago [-]
System 1 isn't even a correct theory in the context of human reasoning, it's just pop-sci nonsense.
samusiam 11 hours ago [-]
I think System 1 is a great term. System 1 is fast, intuitive, and automatic. It describes decision-making that happens without explicit reasoning or deliberation. System 2 is the opposite: it's the more deliberate, "executive functioning" side of cognition -- the part that reasons through a problem before arriving at an answer. That's also what state-of-the-art LLMs do before they respond. Jev doesn’t do that kind of reasoning. It just decides.
SwellJoe 7 hours ago [-]
I don't like anthropomorphizing terminology applied to LLMs, in general, so I kind of object to it on that grounds, rather than whether System 1 means "fast".
fxwin 11 hours ago [-]
so whats the difference between this and a non-reasoning LLM, or just any generic classifier method that necessitates a new term? there is nothing more intuitive or automatic about jev or this than any of the other currently used AI models
aoeusnth1 5 hours ago [-]
The difference is the the joint embedding of actions and state which are fined tuned for certain outcomes. The bulk of the weights are just a forward pass on an LLM (Nemotron).
fxwin 2 hours ago [-]
is that how jev works too? how do these differences make those approaches more "instinctive and emotional"? I know they're different on a technical level, I'm asking which characteristic difference necessitates the use of this loaded term from pop psych
agumonkey 11 hours ago [-]
I wonder if there are research on when system 2 keeps taking over because system 1 has derailed.
vessenes 9 hours ago [-]
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%).
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
kuukyo 8 hours ago [-]
They don't report the pass@1 success rate. They sample multiple solutions from Opus/Fable and CLM decides which one to submit, that's why they get >80%.
abeppu 8 hours ago [-]
Maybe I'm misunderstanding this but when would you ever use it this way? If you're already willing to call Opus/Fable, then isn't the obvious comparison whether Opus/Fable can choose among sampled solutions better or worse than their fast model? If you're willing to pay many seconds for many code samples from a slow model, it's contrived to imagine you care about picking between them in ms.
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
brookman64k 15 hours ago [-]
I tried TypeSafe’s Jev playground. It outputs the model latency and network latency separately. The model latency was 100-200ms in my tests.
rgbrgb 5 hours ago [-]
beautiful. would love to see it on some leaderboard
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?
handzhiev 11 hours ago [-]
Genuine question: Are the use-cases similar to Jev or do you see it applicable for something different? Too dumb to figure it out on my own.
in-silico 18 hours ago [-]
I wonder when work started on this project, and how the public release of Jev played into their timing.
I actually explored this for robotics last year, and CLM grew directly out of that work a few months ago. Feel free to check out my earlier post for more details on how CLM could be applied to robotics:
https://x.com/jackyk02/status/2026368947210289660
I tried running this on a H100 and got 190ms compared to Jev's 170ms. Maybe I set it up wrong?
kevmo314 7 hours ago [-]
It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.
peter_d_sherman 11 hours ago [-]
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%). [...]
A full pre-training run on the Nemotron DQA dataset takes about an hour on a single RTX 4090 GPU.
Most importantly, since states and actions are disaggregated, their embeddings can be cached independently. In settings where the state evolves continuously (e.g., Super Mario) while the action set remains fixed, we only need to recompute the state embedding at each step and can reuse the cached action embeddings. This substantially reduces inference cost, with the efficiency gains becoming increasingly significant as the number of candidate actions and context length grows."
There does definitely seem to be something there with respect to Contrastive Language Models.
They are probably worth studying for people (like myself!) who want to wring the absolute last cycle of local AI training and inferencing performance out of consumer-grade (i.e., not datacenter scale nor cost) hardware...
15 hours ago [-]
0x4139 15 hours ago [-]
"On par with Jev" is Mario, T-Rex, WikiRacing: large-K semantic action matching, the regime where cosine over independently encoded vectors is the right inductive bias. The zero-shot suite does not cover the typed-decision load: date arithmetic, negation chains, policy thresholds.
System 1 is quick and intuitive. System 2 is slow and deliberate.
Compared to models that compose responses with many forward passes, thinking in token space, these models make a quick decision after processing their inputs.
If anyone is in the mood to stomach a paper written by someone who did horrible things with Epstein a few decades later, this is sadly still the best in the biz: https://www.inf.ufsc.br/~mauro.roisenberg/ine6102/leituras/a...
If you prefer geniuses that aren't arguably monsters, this is the core of what Gary Marcus' "neurosymbolic" thing is about.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
This is self contradictory. You can tell a correct answer as you said, by looking at the score. In my $DAYJOB I am making hundreds of RL environments that produce a score for each intermediate state.
Let's give an example: Suppose you ran a Jev that tried to determine when another Jev was wrong about something ...baseball games, let's say. The second Jev would come up with a perfect list of when the first Jev was wrong and when not to believe it. So now you have a second-order system that you believe more than the first-order one.
Only after 100 baseball games, the second-order Jev is only as good as the first one, it just inverted a bunch of games that could have gone either way. So hell, you make a third-order Jev that analyzes the first two...and its results are AMAZING when you look at the historical record! Only, you know, that's what's called overfitting.
Honestly, [edit: Fuck, I just wrote "honestly". I've been brain damaged by you-know-who] the invention of the "noul" is a bit of a giveaway. Imagine what junk bond traders could've done with that in the '80s. Not-not-not-falsy is how we all like our stock picks served up, right?
Reading the raw statistical output of an LLM as if it were an oracular source of truth is literally idol worship and gambling in the purest sense.
Take it from a guy who lost $10k on baseball this season having Claude rewrite my original code to reverse and re-reverse underdogs and favorites; there is no fucking "probability" of anything coming out of an LLM, even if your source of truth isn't an LLM but an evolutionary algorithm you designed yourself. Which would've performed better before the LLM started interpreting ways to bucket it and make up bullshit probabilities around it.
Anyone going down the Jev path is deeply misguided, but will see the light once they realize they have re-invented the magic 8 ball. Or that executive decision maker cube from the 80s with 6 random answers.
Here's your Jev. Note the preponderance of nouls ;)
https://www.ebay.com/itm/146783842141
If you happen to gamble or just watch baseball, you'll see that reality is not so easily reduced to one number. On the most basic level, a single probability number leaves out volatility, without which it's utterly useless for predicting anything other than a sequence it's already trained and fitted for. But as a gambling addict and occasional patron of oracles and gurus, it's very easy and tempting to mistake a clear-cut "probably" for whatever you want to hear. Listening to an LLM's heartbeat for probabilities is just a gambler's fallacy taken out to the 19th degree.
What it would need is a calibration dataset on which to align. There is no calibration in the abstract, only relative to a set of test examples. A model with an uncalibrated output probability can be recalibrated using conformal prediction. You run the model over your calibration examples, get the probabilities.
Assume the new example's answer is y, and calculate its nonconformity score, higher means a worse fit. Count how many calibration examples have a score at least as high as that. Add one to this count, then divide by the total number of calibration examples plus one.
I can tell you though, I'm not alone... a completely degenerate cokehead at my local bar just showed me a baseball prediction app he had Claude build for him on his phone, which looked suspiciously similar to mine. (Mine might be hand coded and backed by a symbolic regression A-Life engine I've been working on since 2005, but so what? Claude has helped me flip most of the predictions for various reasons, to equal "success"). Crude, and yet his app somehow included wind direction forecasts for each inning of every game. This guy is smart as a punter but has probably never used a computer for anything in his life beyond downloading STD test results.
It's a brave new world.
https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow
> "System 1" is fast, instinctive and emotional
this implies more than just "fast", which is precisely why i don't like its present usage
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
[1] https://openai.com/index/clip/
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
https://benchmarkheaven.com/jev-models
https://huggingface.co/spaces/multimodalart/jev-decision-ind...
https://x.com/jackyk02/status/2102947585208258799?s=20
But of course the repo with the TypeSafe compatible API is new: https://github.com/Contrastive-LM/CLMA full pre-training run on the Nemotron DQA dataset takes about an hour on a single RTX 4090 GPU.
Most importantly, since states and actions are disaggregated, their embeddings can be cached independently. In settings where the state evolves continuously (e.g., Super Mario) while the action set remains fixed, we only need to recompute the state embedding at each step and can reuse the cached action embeddings. This substantially reduces inference cost, with the efficiency gains becoming increasingly significant as the number of candidate actions and context length grows."
There does definitely seem to be something there with respect to Contrastive Language Models.
They are probably worth studying for people (like myself!) who want to wring the absolute last cycle of local AI training and inferencing performance out of consumer-grade (i.e., not datacenter scale nor cost) hardware...