A skeptical take on Jev

Aradhye Agarwal

@AradhyeAgarwal

While the whole world is going crazy over Jev, I give you a more skeptical take. I think it's a great model––but its modeling choices cause real failure modes. 1. The way the speed argument is presented in the quoted tweet isn't valid IMO. You can simply treat every row in the output as independent and sample the autoregressive LLM in parallel. 2. The main benefit of autogressive models is enabling increased computation depth with the same amount of model parameters (via CoT). I don't see how that would be possible with Jev. 3. Jev isn't a magic wand that will one-shot any decision task. I can come up with a task where LLMs do better in terms of efficiency: suppose you have 100 choices out of which you have to choose 10. Also suppose someone has already chosen some 10 choices. You have to tell the "goodness" of this choice. With Jev you could naively multiply the probabilities of choosing the options, but this would not work. Making one choice changes the probability distribution of the other options which has to be accounted for. A simple example one of the chosen options is repeated. Jev will give both the options equal probability (due to symmetry) but it's unlikely that this would be the optimal choice (assume repeating options is downright bad). With LLMs however you can prompt it like so: "okay you have 100 options and you have to answer which ones (a total of 10) you would choose in a sorted order." Then gaslight the LLM into thinking it generated the 10 options (in sorted order) by just prefixing those tokens to the assistant channel and then measure the probability of generating that output. This would happen in one-shot. A careful reader might say: "but we can do the same thing with Jev too right?" Yes that's correct, you could have Jev choose one option then conditioned on that choose the second one and so on. Then multiple the probabilities. But the problem is that this will take 10 sequential steps; you can't one-shot it unlike with an LLM where there is no recurrence.