Curious to learn what a “product benchmark” looks like. Is it evals you use to test prompts&#x2F;models? A third party tool?Examples from the wild are a great learning tool, anything you’re able to share is appreciated.

May I ask your internal benchmark ? I&#x27;m building a new set of benchmarks and testing suite for agentic workflows using deepwalker [0]. How do you design your benchmark suite ? would be really cool if you can give more details.[0] <a href="https:&#x2F;&#x2F;deepwalker.xyz" rel="nofollow">https:&#x2F;&#x2F;deepwalker.xyz</a>

Oh wow - I recently tried 3 Pro preview and it was too slow for me.After reading your comment I ran my product benchmark against 2.5 flash, 2.5 pro and 3.0 flash.The results are better AND the response times have stayed the same.
What an insane gain - especially considering the price compared to 2.5 Pro.
I&#x27;m about to get much better results for 1&#x2F;3rd of the price. Not sure what magic Google did here, but would love to hear a more technical deep dive comparing what they do different in Pro and Flash models to achieve such a performance.Also wondering, how did you get early access? I&#x27;m using the Gemini API quite a lot and have a quite nice internal benchmark suite for it, so would love to toy with the new ones as they come out.

I don&#x27;t think tricky niche knowledge is the sweet spot for genai and it likely won&#x27;t be for some time. Instead, it&#x27;s a great replacement for rote tasks where a less than perfect performance is good enough. Transcription, ocr, boilerplate code generation, etc.

So this is an interesting benchmark, because if the answer is actually in the top 3 google results, then my python script that runs a google search, scrapes the top n results and shoves them into a crappy LLM would pass your benchmark too!Which also implies that (for most tasks), most of the weights in a LLM are unnecessary, since they are spent on memorizing the long tail of Common Crawl... but maybe memorizing infinite trivia is not a bug but actually required for the generalization to work? (Humans don&#x27;t have far transfer though... do transformers have it?)

Hi. I am curious what was the benchmark question? Cheers!

I also have my own tricky benchmark that up til now only Deepseek has been able to answer. Gemini 3 Pro was the second. Every other LLM fail horribly. This is the main reason I started looking at G3pro more seriously.

Even the most magical wonderful auto-hammer is gonna be bad at driving in screws. And, in this analogy I can&#x27;t fault you because there are people trying to sell this hammer as a screwdriver. My opinion is that it&#x27;s important to not lose sight of the places where it is useful because of the places where it isn&#x27;t.

I&#x27;m a significant genAI skeptic.I periodically ask them questions about topics that are subtle or tricky, and somewhat niche, that I know a lot about, and find that they frequently provide extremely bad answers. There have been improvements on some topics, but there&#x27;s one benchmark question that I have that just about every model I&#x27;ve tried has completely gotten wrong.Tried it on LMArena recently, got a comparison between Gemini 2.5 flash and a codenamed model that people believe was a preview of Gemini 3 flash. Gemini 2.5 flash got it completely wrong. Gemini 3 flash actually gave a reasonable answer; not quite up to the best human description, but it&#x27;s the first model I&#x27;ve found that actually seems to mostly correctly answer the question.So, it&#x27;s just one data point, but at least for my one fairly niche benchmark problem, Gemini 3 Flash has successfully answered a question that none of the others I&#x27;ve tried have (I haven&#x27;t actually tried Gemini 3 Pro, but I&#x27;d compared various Claude and ChatGPT models, and a few different open weights models).So, guess I need to put together some more benchmark problems, to get a better sample than one, but it&#x27;s at least now passing a &quot;I can find the answer to this in the top 3 hits in a Google search for a niche topic&quot; test better than any of the other models.Still a lot of things I&#x27;m skeptical about in all the LLM hype, but at least they are making some progress in being able to accurately answer a wider range of questions.

Hard to find info but I think the -chat versions of 5.1 and 5.2 (gpt-5.2-chat) are what you&#x27;re looking for. They might just be an alias for the same model with very low reasoning though. I&#x27;ve seen other providers do the same thing, where they offer a reasoning and non reasoning endpoint. Seems to work well enough.

Hardware is a factor here. GPUs are necessarily higher latency than TPUs for equivalent compute on equivalent data. There are lots of other factors here, but latency specifically favours TPUs.The only non-TPU fast models I&#x27;m aware of are things running on Cerebras can be much faster because of their CPUs, and Grok has a super fast mode, but they have a cheat code of ignoring guardrails and making up their own world knowledge.

I had wondered if they run their inference at high batch sizes to get better throughput to keep their inference costs lower.They do have a priority tier at double the cost, but haven&#x27;t seen any benchmarks on how much faster that actually is.The flex tier was an underrated feature in GPT5, batch pricing with a regular API call. GPT5.1 using flex priority is an amazing price&#x2F;intelligence tradeoff for non-latency sensitive applications, without needing to extra plumbing of most batch APIs

Yeah, I&#x27;m surprised that they&#x27;ve been through GT-5.1 and GPT-5.1-Codex and GPT-5.1-Codex-Max and now GPT-5.2 but their most recent mini model is still GPT-5-mini.

&gt; OpenAI made a huge mistake neglecting fast inferencing models.It&#x27;s a lost battle. It&#x27;ll always be cheaper to use an open source model hosted by others like together&#x2F;fireworks&#x2F;deepinfra&#x2F;etc.I&#x27;ve been maining Mistral lately for low latency stuff and the price-quality is hard to beat.

&gt; OpenAI should stop trying to come up with ads and make models that are useful.Turns out becoming a $4 trillion company first with ads (Google), then owning everybody on the AI-front could be the winning strategy.

OpenAI made a huge mistake neglecting fast inferencing models. Their strategy was gpt 5 for everything, which hasn&#x27;t worked out at all. I&#x27;m really not sure what model OpenAI wants me to use for my applications that require lower latency. If I follow their advice in their API docs about which models I should use for faster responses I get told either use GPT 5 low thinking, or replace gpt 5 with gpt 4.1, or switch to the mini model. Now as a developer I&#x27;m doing evals on all three of these combinations. I&#x27;m running my evals on gemini 3 flash right now, and it&#x27;s outperforming gpt5 thinking without thinking. OpenAI should stop trying to come up with ads and make models that are useful.

Can confirm. We at Roblox open sourced a new frontier game eval today, and it&#x27;s beating even Gemini 3 Pro! ( Previous best model ).<a href="https:&#x2F;&#x2F;github.com&#x2F;Roblox&#x2F;open-game-eval&#x2F;blob&#x2F;main&#x2F;LLM_LEADERBOARD.md" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Roblox&#x2F;open-game-eval&#x2F;blob&#x2F;main&#x2F;LLM_LEADE...</a>

On your Omniscience-Index vs. Cost graph, I think your Gemini 3 pro &amp; flash models might be swapped.

Alright so we have more benchmarks including hallucinations and flash doesn&#x27;t do well with that, though generally it beats gemini 3 pro and GPT 5.1 thinking and gpt 5.2 thinking xhigh (but then, sonnet, grok, opus, gemini and 5.1 beat 5.2 xhigh) - everything. Crazy.<a href="https:&#x2F;&#x2F;artificialanalysis.ai&#x2F;evaluations&#x2F;omniscience" rel="nofollow">https:&#x2F;&#x2F;artificialanalysis.ai&#x2F;evaluations&#x2F;omniscience</a>

Lately I was trying ask LLMs to generate SVG pictures, do you have famous pelican on bike created by flash model?

Amazon Set to Waste $10 Billion on OpenAI - <a href="https:&#x2F;&#x2F;finance.yahoo.com&#x2F;news&#x2F;amazon-set-waste-10-billion-151541306.html" rel="nofollow">https:&#x2F;&#x2F;finance.yahoo.com&#x2F;news&#x2F;amazon-set-waste-10-billion-1...</a> - December 17th, 2025

Seeing Sergey Brin back in the trenches makes me think Google is really going to win thisThey always had the best talent, but with Brin at the helm, they also have someone with the organizational heft to drive them towards a single goal

Very soon, because clearly OpenAI is in very serious trouble. They are scaled and have no business model and a competitor that is much better than them at almost everything (ads, hardware, cloud, consumer, scaling).

Oracle&#x27;s stock skyrocketed then took a nosedive. Financial experts warned that companies who bet big on OpenAI like Oracle and Coreweave to pump their stock would go down the drain, and down the drain they went (so far: -65% for Coreweave and nearly -50% of Oracle compared to their OpenAI-hype all-time highs).Markets seems to be in a: &quot;Show me the OpenAI money&quot; mood at the moment.And even financial commentators who don&#x27;t necessarily know a thing about AI can realize that Gemini 3 Pro and now Gemini 3 Flash are giving ChatGPT a run for its money.Oracle and Microsoft have other source of revenues but for those really drinking the OpenAI koolaid, including OpenAI itself, I sure as heck don&#x27;t know what the future holds.My safe bet however is that Google ain&#x27;t going anywhere and shall keep progressing on the AI front at an insane pace.

OpenAI&#x27;s doom was written when Altman (and Nadella) got greedy, threw away the nonprofit mission, and caused the exodus of talent and funding that created Anthropic. If they had stayed nonprofit the rest of the industry could have consolidated their efforts against Google&#x27;s juggernaut. I don&#x27;t understand how they expected to sustain the advantage against Google&#x27;s infinite money machine. With Waymo Google showed that they&#x27;re willing to burn money for decades until they succeed.This story also shows the market corruption of Google&#x27;s monopolies, but a judge recently gave them his stamp of approval so we&#x27;re stuck with it for the foreseeable future.

But you’re forgetting the Jonny Ive hardware device that totally isn’t like that laughable pin badge thing from Humane&#x2F;s

I wonder at what point will everyone who over-invested in OpenAI will regret their decision (expect maybe Nvidia?). Maybe Microsoft doesn&#x27;t need to care, they get to sell their models via Azure.

Yes, 2.5 Flash is extremely cost efficient in my favourite private benchmark: playing text adventures[1]. I&#x27;m looking forward to testing 3.0 Flash later today.[1]: <a href="https:&#x2F;&#x2F;entropicthoughts.com&#x2F;haiku-4-5-playing-text-adventures" rel="nofollow">https:&#x2F;&#x2F;entropicthoughts.com&#x2F;haiku-4-5-playing-text-adventur...</a>

Thanks, having it walk a hardcore SDR signal chain right now --- oh damn it just finished. The blog post makes it clear this isn&#x27;t just some &#x27;lite&#x27; model - you get low latency and cognitive performance. really appreciate you amplifying that.

Interesting. Flash suggests more power to me than Mini. I never use gpt-5-mini in the UI whereas Flash appears to be just as good as Pro just a lot faster.

Fair point. Asked Gemini to suggest alternatives, and it suggested Gemini Velocity, Gemini Atom, Gemini Axiom (and more). I would have liked `Gemini Velocity`.

&gt; Don’t let the “flash” name fool youI think it&#x27;s bad naming on google&#x27;s part. &quot;flash&quot; implies low quality, fast but not good enough. I get less negative feeling looking at &quot;mini&quot; models.

What are you using it for and what were you using before?

Gemini 2.0 flash was good already for some tasks of mine long time ago..

My experience so far- much less reliable. Though it’s been in chat not opencode or antigravity etc. you give it a program and say change it in this way, and it just throws stuff away, changes unrelated stuff etc. completely different quality than pro (or sonnet 4.5 &#x2F; GPT-5.2)

In my own, very anecdotal, experience, Gemini 3 Pro and Flash are both more reliably accurate than GPT 5.x.I have not worked with Sonnet enough to give an opinion there.

How good is it for coding, relative to recent frontier models like GPT 5.x, Sonnet 4.x, etc?

What type of question is your one about testing AI inference time?

Make me think of how every iPhone is the best iPhone yet.Waiting for Apple to say &quot;sorry folks, bad year for iPhone&quot;

I love how every single LLM model release is accompanied by pre-release insiders proclaiming how it’s the best model yet…

Did you try with the grounding tool? Turning it on solved this problem for me.

Cool! I&#x27;ve been using 2.5 flash and it is pretty bad. 1 out of 5 answers it gives will be a lie. Hopefully 3 is better

Can you be more specific on the tasks you’ve found exceptional ?

I agree with this observation. Gemini does feel like code-red for basically every AI company like chatgpt,claude etc. too in my opinion if the underlying model is both fast and cheap and good enoughI hope open source AI models catch up to gemini 3 &#x2F; gemini 3 flash. Or google open sources it but lets be honest that google isnt open sourcing gemini 3 flash and I guess the best bet mostly nowadays in open source is probably glm or deepseek terminus or maybe qwen&#x2F;kimi too.

Coding is basically an edge case for LLMs too.Pretty much every person in the first (and second) world is using AI now, and only small fraction of those people are writing software. This is also reflected in OAI&#x27;s report from a few months ago that found programming to only be 4% of tokens.

I think google is the only one that still produce general knowledge LLM right nowclaude is coding model from the start but GPT is in more and more becoming coding model

Two orders of magnitude would imply that these models cost $28&#x2F;m in and $42&#x2F;m out. Nothing is even close to that.

I struggle to see the incentive to do this, I have similar thoughts for locally run models. It&#x27;s only use case I can imagine is small jobs at scale perhaps something like auto complete integrated into your deployed application, or for extreme privacy, honouring NDA&#x27;s etc.Otherwise, if it&#x27;s a short prompt or answer, SOTA (state of the art) model will be cheap anyway and id it&#x27;s a long prompt&#x2F;answer, it&#x27;s way more likely to be wrong and a lot more time&#x2F;human cost is spent on &quot;checking&#x2F;debugging&quot; any issue or hallucination, so again SOTA is better.

Just to point this out: many of these frontier models cost isn&#x27;t that far away from two orders of magnitude more than what DeepSeek charges. It doesn&#x27;t compare the same, no, but with coaxing I find it to be a pretty capable competent coding model &amp; capable of answering a lot of general queries pretty satisfactorily (but if it&#x27;s a short session, why economize?). $0.28&#x2F;m in, $0.42&#x2F;m out. Opus 4.5 is $5&#x2F;$25 (17x&#x2F;60x).I&#x27;ve been playing around with other models recently (Kimi, GPT Codex, Qwen, others) to try to better appreciate the difference. I knew there was a big price difference, but watching myself feeding dollars into the machine rather than nickles has also founded in me quite the reverse appreciation too.I only assume &quot;if you&#x27;re not getting charged, you are the product&quot; has to be somewhat in play here. But when working on open source code, I don&#x27;t mind.

&gt; it’s more performant than Claude Opus 4.5 or GPT 5.2 extra high...and all of that done without any GPUs as far as i know! [1][1] - <a href="https:&#x2F;&#x2F;www.uncoveralpha.com&#x2F;p&#x2F;the-chip-made-for-the-ai-inference" rel="nofollow">https:&#x2F;&#x2F;www.uncoveralpha.com&#x2F;p&#x2F;the-chip-made-for-the-ai-infe...</a>(tldr: afaik Google trained Gemini 3 entirely on tensor processing units - TPUs)

Should I not let the &quot;Gemini&quot; name fool me either?

Don’t let the “flash” name fool you, this is an amazing model.I have been playing with it for the past few weeks, it’s genuinely my new favorite; it’s so fast and it has such a vast world knowledge that it’s more performant than Claude Opus 4.5 or GPT 5.2 extra high, for a fraction (basically order of magnitude less!!) of the inference time and price

I&#x27;m more curious how Gemini 3 flash lite performs&#x2F;is priced when it comes out. Because it may be that for most non coding tasks the distinction isn&#x27;t between pro and flash but between flash and flash lite.

Thanks that was a great breakup of cost. I just assumed before that it was the same pricing. The pricing probably comes from the confidence and the buzz around Gemini 3.0 as one of the best performing models. But competetion is hot in the area and it&#x27;s not too far where we get similar performing models for cheaper price.

<a href="https:&#x2F;&#x2F;openrouter.ai&#x2F;google&#x2F;gemini-3-flash-preview" rel="nofollow">https:&#x2F;&#x2F;openrouter.ai&#x2F;google&#x2F;gemini-3-flash-preview</a>

Token usage also needs to be factored in specifically when thinking is enabled, these newer models find more difficult problems easier and use less tokens to solve.

flash is closer to sonnet than gpt minis though

For comparison, GPT-5 mini is $0.25&#x2F;M for input and $2.00&#x2F;M for output, so double the price for input and 50% higher for output.

The price increase sucks, but you really do get a whole lot more. They also had the &quot;Flash Lite&quot; series, 2.5 Flash Lite is 0.10&#x2F;M, hopefully we see something like 3.0 Flash Lite for .20-.25.

Mostly at the time of release except for 1.5 Flash which got a price drop in Aug 2024.Google has been discontinuing older models after several months of transition period so I would expect the same for the 2.5 models. But that process only starts when the release version of 3 models is out (pro and flash are in preview right now).

Are these the current prices or the prices at the time the models were released?

There are plenty. But it&#x27;s not the comparison you want to be making. There is too much variability between the number of tokens used for a single response, especially once reasoning models became a thing. And it gets even worse when you put the models into a variable length output loop.You really need to look at the cost per task. artificialanalysis.ai has a good composite score, measures the cost of running all the benchmarks, and has 2d a intelligence vs. cost graph.

<a href="https:&#x2F;&#x2F;www.helicone.ai&#x2F;llm-cost">https:&#x2F;&#x2F;www.helicone.ai&#x2F;llm-cost</a>Tried a lot of them and settled on this one, they update instantly on model release and having all models on one page is the best UX.

<a href="https:&#x2F;&#x2F;openrouter.ai&#x2F;models" rel="nofollow">https:&#x2F;&#x2F;openrouter.ai&#x2F;models</a>

<a href="https:&#x2F;&#x2F;www.llm-prices.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.llm-prices.com&#x2F;</a>

is there a website where i can compare openai, anthropic and gemini models on cost&#x2F;token ?

This is awesome. No preview release either, which is great to production.They are pushing the prices higher with each release though:
API pricing is up to $0.5&#x2F;M for input and $3&#x2F;M for outputFor comparison:Gemini 3.0 Flash: $0.50&#x2F;M for input and $3.00&#x2F;M for outputGemini 2.5 Flash: $0.30&#x2F;M for input and $2.50&#x2F;M for outputGemini 2.0 Flash: $0.15&#x2F;M for input and $0.60&#x2F;M for outputGemini 1.5 Flash: $0.075&#x2F;M for input and $0.30&#x2F;M for output (after price drop)Gemini 3.0 Pro: $2.00&#x2F;M for input and $12&#x2F;M for outputGemini 2.5 Pro: $1.25&#x2F;M for input and $10&#x2F;M for outputGemini 1.5 Pro: $1.25&#x2F;M for input and $5&#x2F;M for outputI think image input pricing went up even more.Correction: It is a preview model...

That&#x27;s too bad. Apple&#x27;s most interesting value proposition is running local inference with big privacy promises. They wouldn&#x27;t need to be the highest performer to offer something a lot of people might want.

Apple Intelligence is going to be Gemini <a href="https:&#x2F;&#x2F;www.macrumors.com&#x2F;2025&#x2F;11&#x2F;05&#x2F;apple-siri-google-gemini-partnership&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.macrumors.com&#x2F;2025&#x2F;11&#x2F;05&#x2F;apple-siri-google-gemin...</a>

Yeah - agree, Anthropic much better for coding. I&#x27;m more thinking about the &#x27;average chat user&#x27; (the larger potential userbase), most of whom are on chatgpt.

Knowing Googles MO, its most likely not the model but their harness system that&#x27;s the issue. God they are so bad at their UI and agentic coding harnesses...

Pulling ahead? Depends on the usecase I guess. 3 turns into a very basic Gemini-CLI session and Gemini 3 Pro has already messed up a simple `Edit` tool-call.
And it&#x27;s awfully slow. In 27 minutes it did 17 tool calls, and only managed to modify 2 files. Meanwhile Claude-Code flies through the same task in 5 minutes.

Analyse e-mails&#x2F;text&#x2F;music&#x2F;videos, edit photos, summarization, etc.

I desperately want to be able to real-time dictate actions to take on my phone.Stuff like:&quot;Open Chrome, new tab, search for xyz, scroll down, third result, copy the second paragraph, open whatsapp, hit back button, open group chat with friends, paste what we copied and send, send a follow-up laughing tears emoji, go back to chrome and close out that tab&quot;All while being able to just quickly glance at my phone. There is already a tool like this, but I want the parsing&#x2F;understanding of an LLM and super fast response times.

What will you use the ai in the phone to do for you? I can understand tablets and smart glasses being able to leverage smol AI much better than a phone which is reliant on apps for most of the work.

Feels like Google is really pulling ahead of the pack here. A model that is cheap, fast and good, combined with Android and gsuite integration seems like such powerful combination.Presumably a big motivation for them is to be first to get something good and cheap enough they can serve to every Android device, ahead of whatever the OpenAI&#x2F;Jony Ive hardware project will be, and way ahead of Apple Intelligence. Speaking for myself, I would pay quite a lot for truly &#x27;AI first&#x27; phone that actually worked.

That has been my experience. Primarily because it is allowed to expend far more test-time tokens than Gemini 3.0 Pro to solve the same prompt.

Chatgpt 5.2 thinking is significantly better quality for most knowledge work, but it trades off in speed.

Fair. No benchmark is perfect.I do pay special attention to what the most negative comments say (which in this case are unusually positive). And people discussing performance on their own personal benchmarks.

The brand bias is heavily against Google, not in Googles favor

No offense, but that seems like a poor benchmark. These initial vibe checks are easily swayed by personal brand biases.

This model is breaking records on my benchmark of choice, which is &#x27;the fraction of Hacker News comments that are positive.&#x27; Even people who avoid Google products on principle are impressed. Hardly anyone is arguing that ChatGPT is better in any respect (except brand recognition).

Yes, but the 3.0 Flash is cheaper, faster and better than 2.5 Pro.So if 2.5 Pro was good for your usecase, you just got a better model for about 1&#x2F;3rd of the price, but might hurt the wallet a bit more if you use 2.5 Flash currently and want an upgrade - which is fair tbh.

I think it&#x27;s good, they&#x27;re raising the size (and price) of flash a bit and trying to position Flash as an actually useful coding &#x2F; reasoning model. There&#x27;s always lite for people who want dirt cheap prices and don&#x27;t care about quality at all.

Nvidia released Nemotron 3 nano recently and I think it fits your requirements for an OSS model:
<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;nvidia&#x2F;NVIDIA-Nemotron-3-Nano-30B-A3B-BF16" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;nvidia&#x2F;NVIDIA-Nemotron-3-Nano-30B-A3B...</a>It&#x27;s extremely fast on good hardware, quite smart, and can support up to 1m context with reasonable accuracy

For my apps evals Gemini flash and grok 4 fast are the only ones worth using. I&#x27;d love for an open weights model to compete in this arena but I haven&#x27;t found one.

This one is more powerful than openai models, including gpt 5.2 (which is worse on various benchmarks than 5.1 which is worse than 5.1, and that&#x27;s where 5.2 was using XHIGH, whiulst the others were on high eg: <a href="https:&#x2F;&#x2F;youtu.be&#x2F;4p73Uu_jZ10?si=x1gZopegCacznUDA&amp;t=582" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;4p73Uu_jZ10?si=x1gZopegCacznUDA&amp;t=582</a> )<a href="https:&#x2F;&#x2F;epoch.ai&#x2F;benchmarks&#x2F;simplebench" rel="nofollow">https:&#x2F;&#x2F;epoch.ai&#x2F;benchmarks&#x2F;simplebench</a>

Sure, but for simple tasks that require a large context window, aka the typical usecase for 2.0 flash, it&#x27;s still significantly more expensive.

cost of e2e task resolution should be cheaper, even if single inference cost is higher, you need fewer loops to solve a problem now

These flash models keep getting more expensive with every release.Is there an OSS model that&#x27;s better than 2.0 flash with similar pricing, speed and a 1m context window?Edit: this is not the typical flash model, it&#x27;s actually an insane value if the benchmarks match real world usage.&gt; Gemini 3 Flash achieves a score of 78%, outperforming not only the 2.5 series, but also Gemini 3 Pro. It strikes an ideal balance for agentic coding, production-ready systems and responsive interactive applications.The replacement for old flash models will be probably the 3.0 flash lite then.

Gemini 2.5 was a full broadside on OpenAI&#x27;s ship.After Gemini 3.0 the OpenAI damage control crews all drowned.Not only is it vastly better, it&#x27;s also free.I find this particular benchmark to be in agreement with my experiences: <a href="https:&#x2F;&#x2F;simple-bench.com" rel="nofollow">https:&#x2F;&#x2F;simple-bench.com</a>

What if you also number the legs, but with an error like: 1,2,3,5,6. Or 1,2,3, ,4.

So gemini 3 flash (non thinking) is now the first model to get 50% on my &quot;count the dog legs&quot; image test.Gemini 3 pro got 20%, and everyone else has gotten 0%. I saw benchmarks showing 3 flash is almost trading blows with 3 pro, so I decided to try it.Basically it is an image showing a dog with 5 legs, an extra one photoshopped onto it&#x27;s torso. Every models counts 4, and gemini 3 pro, while also counting 4, said the dog had a &quot;large male anatomy&quot;. However it failed a follow-up saying 4 again.3 flash counted 5 legs on the same image, however I added distinct a &quot;tattoo&quot; to each leg as an assist. These tattoos didn&#x27;t help 3 pro or other models.So it is the first out of all the models I have tested to count 5 legs on the &quot;tattooed legs&quot; image. It still counted only 4 legs on the image without the tattoos. I&#x27;ll give it 1&#x2F;2 credit.

Not OP, but I feel the same way. Cost is just one of the factor. I&#x27;m used to Claude Code UX, my CLAUDE.md works well with my workflow too. Unless there&#x27;s any significant improvement, changing to new models every few months is going to hurt me more.

I think a big part of the switching cost is the cost of learning a different model&#x27;s nuances. Having good intuition for what works&#x2F;doesn&#x27;t, how to write effective prompts, etc.Maybe someday future models will all behave similarly given the same prompt, but we&#x27;re not quite there yet

Because some people are restricted by company policy to only use providers with which they have a legally binding agreement to not use their chats as training data.

Why wouldn&#x27;t you switch? The cost to switch is near zero for me. Some tools have built in model selectors. Direct CLI&#x2F;IDE plug-ins practically the same UI.

This has been my experience exactly. Even over just the last few weeks I’ve noticed a dramatic drop in having to undo what the agents have done.

For me, the last wave of models finally started delivering on their agentic coding promises.

I asked a similar question yesterday:<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=46290797">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=46290797</a>

But for me the previous models were routinely wrong time wasters that overall added no speed increase taking the lottery of whether they&#x27;d be correct into account.

If Opus is one-size-fits-all, then why Claude keeps the other series? (rethorical).Opus and Sonnet are slower than Haiku. For lots of less sophisticated tasks, you benefit from the speed.All vendors do this. You need smaller models that you can rapid-fire for lots of other reasons than vibe coding.Personally, I actually use more smaller models than the sophisticated ones. Lots of small automations.

Correct. Opus 4.5 &#x27;solved&#x27; software engineering. What more do I need? Businesses need uncapped intelligence, and that is a very high bar. Individuals often don&#x27;t.

The internet has already tried this, for about a few decades. The garbage is in the corpus; it gets weighted as such

&gt;a bunch of internet pages containing things that are blatantly wrongSo Reddit?I’d imagine the AI companies have all the “pre AI internet” data they scraped very carefully catalogued.

I just can&#x27;t stop thinking though about the vulnerability of training dataYou say good enough. Great, but what if I as a malicious person were to just make a bunch of internet pages containing things that are blatantly wrong, to trick LLMs?

Even before this release the tools (for me: Claude Code and Gemini for other stuff) reached a &quot;good enough&quot; plateau that means any other company is going to have a hard time making me (I think soon most users) want to switch. Unless a new release from a different company has a real paradigm shift, they&#x27;re simply sufficient. This was not true in 2023&#x2F;2024 IMO.With this release the &quot;good enough&quot; and &quot;cheap enough&quot; intersect so hard that I wonder if this is an existential threat to those other companies.

I’m amazed by how much Gemini 3 flash hallucinates; it performs poorly in that metric (along with lots of other models). In the Hallucination Rate vs. AA-Omniscience Index chart, it’s not in the most desirable quadrant; GPT-5.1 (high), opus 4.5 and 4.5 haiku are.Can someone explain how Gemini 3 pro&#x2F;flash then do so well then in the overall Omniscience: Knowledge and Hallucination Benchmark?

Also<a href="https:&#x2F;&#x2F;artificialanalysis.ai&#x2F;evaluations&#x2F;omniscience" rel="nofollow">https:&#x2F;&#x2F;artificialanalysis.ai&#x2F;evaluations&#x2F;omniscience</a>Prepare to be amazed

&gt; reflects either training on the test set or some sort of cracked way to pack a ton of parametric knowledge into a Flash ModelThat&#x27;s what MoE is for. It might be that with their TPUs, they can afford lots of params, just so long as the activated subset for each token is small enough to maintain throughput.

This will be fantastic for voice. I presume Apple will use it

Or could it be that it&#x27;s using tool calls in reasoning (e.g. a google search)?

&gt;or some sort of cracked way to pack a ton of parametric knowledge into a Flash Model.More experts with a lower pertentage of active ones -&gt; more sparsity.

It has a SimpleQA score of 69%, a benchmark that tests knowledge on extremely niche facts, that&#x27;s actually ridiculously high (Gemini 2.5 *Pro* had 55%) and reflects either training on the test set or some sort of cracked way to pack a ton of parametric knowledge into a Flash Model.I&#x27;m speculating but Google might have figured out some training magic trick to balance out the information storage in model capacity. That or this flash model has huge number of parameters or something.

“And then imagine Google designing silicon that doesn’t trail the industry. While you are there we may as well start to imagine Google figures out how to support a product lifecycle that isn’t AdSense”Google is great on the data science alone, every thing else is an after thought

&quot;Now, imagine for a moment they had also vertically integrated the hardware to do this.&quot;Then you realise you aren&#x27;t imagining it.

It&#x27;s the only model provider that has offered a decent deal to students: a full year of google ai pro.Granted, this doesn&#x27;t give api access, only what google calls their &quot;consumer ai products&quot;, but it makes a huge difference when chatgpt only allows a handful of document uploads and deep research queries per day.

on aistudio the free tier limits on all models are decent

&gt; think about what would be most terrifying to Anthropic and OpenAIThe most terrifying thing would be Google expanding its free tiers.

I think about what would be most terrifying to Anthropic and OpenAI i.e. The absolute scariest thing that Google could do. I think this is it: Release low latency, low priced models with high cognitive performance and big context window, especially in the coding space because that is direct, immediate, very high ROI for the customer.Now, imagine for a moment they had also vertically integrated the hardware to do this.

Quick pricing comparison: <a href="https:&#x2F;&#x2F;www.llm-prices.com&#x2F;#it=100000&amp;ot=10000&amp;sel=gemini-3-flash-preview%2Cgemini-3-pro-preview%2Cgemini-3-pro-preview-200k%2Cgpt-5.2%2Cclaude-opus-4-5%2Cclaude-sonnet-4.5%2Cclaude-4.5-haiku%2Cgpt-5-mini" rel="nofollow">https:&#x2F;&#x2F;www.llm-prices.com&#x2F;#it=100000&amp;ot=10000&amp;sel=gemini-3-...</a>It&#x27;s 1&#x2F;4 the price of Gemini 3 Pro ≤200k and 1&#x2F;8 the price of Gemini 3 Pro &gt;200k - notable that the new Flash model doesn’t have a price increase after that 200,000 token point.It’s also twice the price of GPT-5 Mini for input, half the price of Claude 4.5 Haiku.

It&#x27;s this, yes: <a href="https:&#x2F;&#x2F;x.com&#x2F;joshwoodward&#x2F;status&#x2F;2001350002975850520" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;joshwoodward&#x2F;status&#x2F;2001350002975850520</a>&gt;Fast = 3 Flash&gt;Thinking = 3 Flash (with thinking)&gt;Pro = 3 Pro (with thinking)

I think:Fast = Gemini 3 Flash without thinking (or very low thinking budget)Thinking = Gemini 3 flash with high thinking budgetPro = Gemini 3 Pro with thinking

It seems:<pre><code> - &quot;Thinking&quot; is Gemini 3 Flash with higher &quot;thinking_level&quot;
 - Prop is Gemini 3 Pro. It doesn&#x27;t mention &quot;thinking_level&quot; but I assume it is set to high-ish.</code></pre>

This is what every agentic coding tool does. You can try it yourself right now with the Gemini CLI, OpenCode, or 20 other tools.

I don&#x27;t think we&#x27;ll get genuine AGI without long-term memory, specifically in the form of weight adjustment rather than just LoRAs or longer and longer contexts. When the model gets something wrong and we tell it &quot;That&#x27;s wrong, here&#x27;s the right answer,&quot; it needs to remember that.Which obviously opens up a can of worms regarding who should have authority to supply the &quot;right answer,&quot; but still... lacking the core capability, AGI isn&#x27;t something we can talk about yet.LLMs will be a part of AGI, I&#x27;m sure, but they are insufficient to get us there on their own. A big step forward but probably far from the last.

AGI is hard but we can solve most tasks with artificial stupidity in an `until done`.

Advanced reasoning LLM&#x27;s simulate many parts of AGI and feel really smart, but fall short in many critical ways.- An AGI wouldn&#x27;t hallucinate, it would be consistent, reliable and aware of its own limitations- An AGI wouldn&#x27;t need extensive re-training, human reinforced training, model updates. It would be capable of true self-learning &#x2F; self-training in real time.- An AGI would demonstrate real genuine understanding and mental modeling, not pattern matching over correlations- It would demonstrate agency and motivation, not be purely reactive to prompting- It would have persistent integrated memory. LLM&#x27;s are stateless and driven by the current context.- It should even demonstrate consciousness.And more. I agree that what&#x27;ve we&#x27;ve designed is truly impressive and simulates intelligence at a really high level. But true AGI is far more advanced.

Really stupid question: How is Gemini-like &#x27;thinking&#x27; separate from artificial general intelligence (AGI)?When I ask Gemini 3 Flash this question, the answer is vague but agency comes up a lot. Gemini thinking is always triggered by a query.This seems like a higher-level programming issue to me. Turn it into a loop. Keep the context. Those two things make it costly for sure. But does it make it an AGI? Surely Google has tried this?

Does anyone else understand what the difference is between Gemini 3 &#x27;Thinking&#x27; and &#x27;Pro&#x27;? Thinking &quot;Solves complex problems&quot; and Pro &quot;Thinks longer for advanced math &amp; code&quot;.I assume that these are just different reasoning levels for Gemini 3, but I can&#x27;t even find mention of there being 2 versions anywhere, and the API doesn&#x27;t even mention the Thinking-Pro dichotomy.

For my personal usage of ai-studio, I had to use autohotkey to record and replay my mouse deleting my old chats. I thought about cooking up a browser extension, but never got around to it.

This is the #1 thing that keeps me from going all in on Gemini.Their retention controls for both consumer and business suck. It’s the worst of any of the leaders.

My main issue with Gemini is that business accounts can&#x27;t delete individual conversations. You can only enable or disable Gemini, or set a retention period (3 months minimum), but there&#x27;s no way to delete specific chats. I&#x27;m a paying customer, prices keep going up, and yet this very basic feature is still missing.

Looking at this they are:<a href="https:&#x2F;&#x2F;artificialanalysis.ai&#x2F;evaluations&#x2F;omniscience" rel="nofollow">https:&#x2F;&#x2F;artificialanalysis.ai&#x2F;evaluations&#x2F;omniscience</a><a href="https:&#x2F;&#x2F;youtu.be&#x2F;4p73Uu_jZ10?si=x1gZopegCacznUDA&amp;t=582" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;4p73Uu_jZ10?si=x1gZopegCacznUDA&amp;t=582</a>

They have been for a while. Had first mover advantage that kept them in the lead but it&#x27;s not anything others couldn&#x27;t throw money at, and catch up eventually. I remember when not so long ago everyone was talking how Google lost AI race, and now it feels like they&#x27;re chasing Anthropic

That’s funny, I’ve had the exact opposite experience. Gemini starts every answer to a coding question with, “you have hit upon a fundamental insight in zyx”. ChatGPT usually starts with, “the short answer? Xyz.”

How do I get Gemini to be more proactive in finding&#x2F;double-checking itself against new world information and doing searches?For that reason I still find chatgpt way better for me, many things I ask it first goes off to do online research and has up to date information - which is surprising as you would expect Google to be way better at this.
For example, was asking Gemini 3 Pro recently about how to do something with a “RTX 6000 Blackwell 96GB” card, and it told me this card doesn’t exist and that I probably meant the rtx 6000 ada… Or just today I asked about something on macOS 26.2, and it told me to be cautious as it’s a beta release (it’s not).
Whereas with chatgpt I trust the final output more since it very often goes to find live sources and info.

I’ve fully switched over to Gemini now. It seems significantly more useful, and is less of an automatic glaze machine that just restates your question and how smart you are for asking it.

I don&#x27;t want to say OpenAI is toast for general chat AI, but it sure looks like they are toast.

Why would you use OpenRouter rather than some local proxy like LiteLLM? I don&#x27;t see the point of sharing data with more third parties and paying for the privilege.Not to mention that for coding, it&#x27;s usually more cost efficient to get whatever subscription the specific model provider offers.

I have used OpenRouter before but in this case I was trying to use it like Claude Code (agentic coding with a simple fixed monthly subscription). I don&#x27;t want to pay per use via direct APIs as I am afraid it might have surprising bills. My point was, why Google makes it so damn hard even for paid subscriptions where it was supposed to work.

Have you tried OpenRouter (<a href="https:&#x2F;&#x2F;openrouter.ai" rel="nofollow">https:&#x2F;&#x2F;openrouter.ai</a>)? I’ve been happy using it as a unified api provider with great model coverage (including Google, Anthropic, OpenAI, Grok, and the major open models). They charge 5% on top of each model’s api costs, but I think it’s worth it to have one centralized place to insert my money and monitor my usage. I like being able to switch out models without having to change my tools, and I like being able to easily head-to-head compare claude&#x2F;gemini&#x2F;gpt when I get stuck on a tricky problem.Then you just have to find a coding tool that works with OpenRouter. Afaik claude&#x2F;codex&#x2F;cursor don’t, at least not without weird hacks, but various of the OSS tools do — cline, roo code, opencode, etc. I recently started using opencode (<a href="https:&#x2F;&#x2F;github.com&#x2F;sst&#x2F;opencode" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;sst&#x2F;opencode</a>), which is like an open version of claude code, and I’ve been quite happy with it. It’s a newer project so There Will Be Bugs, but the devs are very active and responsive to issues and PRs.

Have you tried Google Antigravity? I use that and GitHub Copilot when I want to use Gemini for coding tasks.

use cursor. it allows you to choose any model to use.

Only if I could figure out how to use it. I have been using Claude Code and enjoy it. I sometimes also try Codex which is also not bad.Trying to use Gemini cli is such a pain. I bought GDP Premium and configured GCP, setup environment variables, enabled preview features in cli and did all the dance around it and it won&#x27;t let me use gemini 3. Why the hell I am even trying so hard?

Yes, that&#x27;s very good because it&#x27;s my main use case for Flash; queries depending on world knowledge. Not science or engineering problems, but think you&#x27;d ask someone that has a really broad knowledge about things and can give quick and straightforward answers.

Glad to see big improvement in the SimpleQA Verified benchmark (28-&gt;69%), which is meant to measure factuality (built-in, i.e. without adding grounding resources). That&#x27;s one benchmark where all models seemed to have low scores until recently. Can&#x27;t wait to see a model go over 90%... then will be years till the competition is over number of 9s in such a factuality benchmark, but that&#x27;d be glorious.

Yes I tried it with minimal and it&#x27;s roughly 3 seconds for prompts that take flash 2.5 1 second.On that note it would be nice to get these benchmark numbers based on the different reasoning settings.

Did you try setting thinkingLevel to minimal?thinkingConfig: {
 thinkingLevel: &quot;low&quot;,
}More about it here <a href="https:&#x2F;&#x2F;ai.google.dev&#x2F;gemini-api&#x2F;docs&#x2F;gemini-3#new_api_features_in_gemini_3" rel="nofollow">https:&#x2F;&#x2F;ai.google.dev&#x2F;gemini-api&#x2F;docs&#x2F;gemini-3#new_api_featu...</a>

That&#x27;s more of a flash-lite thing now, I believe

This might also have to do with it being a preview, and only available on the global region?

&gt;You cannot disable thinking for Gemini 3 Pro. Gemini 3 Flash also does not support full thinking-off, but the minimal setting means the model likely will not think (though it still potentially can). If you don&#x27;t specify a thinking level, Gemini will use the Gemini 3 models&#x27; default dynamic thinking level, &quot;high&quot;.<a href="https:&#x2F;&#x2F;ai.google.dev&#x2F;gemini-api&#x2F;docs&#x2F;thinking#levels" rel="nofollow">https:&#x2F;&#x2F;ai.google.dev&#x2F;gemini-api&#x2F;docs&#x2F;thinking#levels</a>

You can still set thinking budget to 0 to completely disable reasoning, or set thinking level to minimal or low.

It&#x27;s a cool release, but if someone on the google team reads that:
flash 2.5 is awesome in terms of latency and total response time without reasoning. In quick tests this model seems to be 2x slower. So for certain use cases like quick one-token classification flash 2.5 is still the better model.
Please don&#x27;t stop optimizing for that!

Yes, but also most of the increase in 3 Flash is in the input context price, which isn&#x27;t affected by reasoning.

Calculating price increases is made more complex by the difference in token usage. From <a href="https:&#x2F;&#x2F;blog.google&#x2F;products&#x2F;gemini&#x2F;gemini-3-flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.google&#x2F;products&#x2F;gemini&#x2F;gemini-3-flash&#x2F;</a> :&gt; Gemini 3 Flash is able to modulate how much it thinks. It may think longer for more complex use cases, but it also uses 30% fewer tokens on average than 2.5 Pro.

Pricing is $0.5 &#x2F; $3 per million input &#x2F; output tokens. 2.5 Flash was $0.3 &#x2F; $2.5. That&#x27;s 66% increase in input tokens and 20% increase in output token pricing.For comparison, from 2.5 Pro ($1.25 &#x2F; $10) to 3 Pro ($2 &#x2F; $12), there was 60% increase in input tokens and 20% increase in output tokens pricing.

Google actually does something similar for major releases - they publish a dedicated collection page with all related links.For example, the Gemini 3 Pro collection: <a href="https:&#x2F;&#x2F;blog.google&#x2F;products&#x2F;gemini&#x2F;gemini-3-collection&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.google&#x2F;products&#x2F;gemini&#x2F;gemini-3-collection&#x2F;</a>But having everything linked at the bottom of the announcement post itself would be really great too!

For anyone from the Gemini team reading this: these links should all be prominent in the announcement posts. I always have to hunt around for them!

Documentation for Gemini 3 Flash in particular: <a href="https:&#x2F;&#x2F;ai.google.dev&#x2F;gemini-api&#x2F;docs&#x2F;gemini-3" rel="nofollow">https:&#x2F;&#x2F;ai.google.dev&#x2F;gemini-api&#x2F;docs&#x2F;gemini-3</a>

Deepmind Page: <a href="https:&#x2F;&#x2F;deepmind.google&#x2F;models&#x2F;gemini&#x2F;flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;deepmind.google&#x2F;models&#x2F;gemini&#x2F;flash&#x2F;</a>Developer Blog: <a href="https:&#x2F;&#x2F;blog.google&#x2F;technology&#x2F;developers&#x2F;build-with-gemini-3-flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.google&#x2F;technology&#x2F;developers&#x2F;build-with-gemini-...</a>Model Card [pdf]: <a href="https:&#x2F;&#x2F;deepmind.google&#x2F;models&#x2F;model-cards&#x2F;gemini-3-flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;deepmind.google&#x2F;models&#x2F;model-cards&#x2F;gemini-3-flash&#x2F;</a>Gemini 3 Flash in Search AI mode: <a href="https:&#x2F;&#x2F;blog.google&#x2F;products&#x2F;search&#x2F;google-ai-mode-update-gemini-3-flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.google&#x2F;products&#x2F;search&#x2F;google-ai-mode-update-ge...</a>

Sonnet 3.5 might have been better than opus 3. That&#x27;s my recollection anyhow

Wild how this beats 2.5 Pro in every single benchmark. Don&#x27;t think this was true for Haiku 4.5 vs Sonnet 3.5.

Omg, it was so frustrating to say:Summarize recent working arxiv urlAnd then it tells me the date is from the future and it simply refuses to fetch the URL.

I wonder if this suffers from the same issue as 3 Pro, that it frequently &quot;thinks&quot; for a long time about date incongruity, insisting that it is 2024, and that information it receives must be incorrect or hypothetical.Just avoiding&#x2F;fixing that would probably speed up a good chunk of my own queries.

Google appears to be changing what flash is “meant for” with this release - the capability it has along with the thinking budgets make it superior to previous Pro models in both outcome and speed. The likely-soon-coming flash-lite will fit right in to where flash used to be - cheap and fast.

Since it now includes 4 thinking levels (minimal-high) I&#x27;d really appreciate if we got some benchmarks across the whole sweep (and not just what&#x27;s presumably high).Flash is meant to be a model for lower cost, latency-sensitive tasks. Long thinking times will both make TTFT &gt;&gt; 10s (often unacceptable) and also won&#x27;t really be that cheap?

They didn&#x27;t put Opus 4.5 on the model card to compare

Looks like a good workhorse model, like I felt 2.5 Flash also was at its time of launch. I hope I can build confidence with it because it&#x27;ll be good to offload Pro costs&#x2F;limits as well of course always nice with speed for more basic coding or queries. I&#x27;m impressed and curious about the recent extreme gains on ARC-AGI-2 from 3 Pro, GPT-5.1 and now even 3 Flash.

Yes! Gemini 3 pro is significantly slower than opus (surprisingly) , and prefer opus&#x27; output.Might be using flash for my MCP research&#x2F;transcriber&#x2F;minor tasks modl over haiku, now, though (will test of course)

$100 Claude max is the best subscription I’ve ever had.Well worth every penny now

I will have to try that. Cursor bill got pretty high with Opus 4.5. Never considered opus before the 4.5 price drop but now it&#x27;s hard to change... :)

Ok, I was a bit addicted to Opus 4.5 and was starting to feel like there&#x27;s nothing like it.Turns out Gemini 3 Flash is pretty close. The Gemini CLI is not as good but the model more than makes up for it.The weird part is Gemini 3 Pro is nowhere as good an experience. Maybe because its just so slow.

OAI also got talent mined. Their top intellectual leaders left after fight with sama, then Meta took a bunch of their mid-senior talent, and Google had the opposite. They brought Noam and Sergey back.

Yeah the only thing standing in Google&#x27;s way is Google. And it&#x27;s the easy stuff, like sensible billing models, easy to use docs and consoles that make sense and don&#x27;t require 20 hours to learn&#x2F;navigate, and then just the slew of bugs in Gemini CLI that are basic usability and model API interaction things. The only differentiator that OpenAI still has is polish.Edit: And just to add an example: openAI&#x27;s Codex CLI billing is easy for me. I just sign up for the base package, and then add extra credits which I automatically use once I&#x27;m through my weekly allowance. With Gemini CLI I&#x27;m using my oauth account, and then having to rotate API keys once I&#x27;ve used that up.Also, Gemini CLI loves spewing out its own chain of thought when it gets into a weird state.Also Gemini CLI has an insane bias to action that is almost insurmountable. DO NOT START THE NEXT STAGE still has it starting the next stage.Also Gemini CLI has been terrible at visibility on what it&#x27;s actually doing at each step - although that seems a bit improved with this new model today.

&quot;At this point in time I start to believe OAI is very much behind on the models race and it can&#x27;t be reversed&quot;This has been true for at least 4 months and yeah, based on how these things scale and also Google&#x27;s capital + in-house hardware advantages, it&#x27;s probably insurmountable.

I&#x27;ve noticed when it comes to evaluating AI models, most people simply don&#x27;t ask difficult enough questions. So everything is good enough, and the preference comes down to speed and style.It&#x27;s when it becomes difficult, like in the coding case that you mentioned, that we can see the OpenAI still has the lead. The same is true for the image model, prompt adherence is significantly better than Nano Banana. Especially at more complex queries.

I&#x27;m actually liking 5.2 in Codex. It&#x27;s able to take my instructions, do a good job at planning out the implementation, and will ask me relevant questions around interactions and functionality. It also gives me more tokens than Claude for the same price. Now, I&#x27;m trying to white label something that I made in Figma so my use case is a lot different from the average person on this site, but so far it&#x27;s my go to and I don&#x27;t see any reason at this time to switch.

For average consumers, I think very much yes, and this is where OpenAI&#x27;s brand recognition shines.But for anyone using LLM&#x27;s to help speed up academic literature reviews where every detail matters, or coding where every detail matters, or anything technical where every detail matters -- the differences very much matter. And benchmarks serve just to confirm your personal experience anyways, as the differences between models becomes extremely apparent when you&#x27;re working in a niche sub-subfield and one model is showing glaring informational or logical errors and another mostly gets it right.And then there&#x27;s a strong possibility that as experts start to say &quot;I always trust &lt;LLM name&gt; more&quot;, that halo effect spreads to ordinary consumers who can&#x27;t tell the difference themselves but want to make sure they use &quot;the best&quot; -- at least for their homework. (For their AI boyfriends and girlfriends, other metrics are probably at play...)

Google biggest advantage over time will be costs. They have their own hardware which they can and will optimise for their LLMS. And Google has experience of getting market share over time by giving better results, performance or space. ie gmail vs hotmail&#x2F;yahoo. Chrome vs IE&#x2F;Firefox. So don&#x27;t discount them if the quality is better they will get ahead over time.

That might be true for a narrow definition of chatbots, but they aren&#x27;t going to survive on name recognition if their models are inferior in the medium term. Right now, &quot;agents&quot; are only really useful for coding, but when they start to be adopted for more mainstream tasks, people will migrate to the tools that actually work first.

this. I don&#x27;t know any non-tech people who use anything other than chatgpt. On a similar note, I&#x27;ve wondered why Amazon doesn&#x27;t make a chatgpt-like app with their latest Alexa+ makeover, seems like a missed opportunity. The Alexa app has a feature to talk to the LLM in chat mode, but the overall app is geared towards managing devices.

I doubt anyone I know who is using llms outside of work knows that there are benchmark tests for these models.

This is why both google and microsoft are pushing Gemini and Copilot in everyone&#x27;s face.

Is there a &quot;good enough&quot; endgame for LLMs and AI where benchmarks stop mattering because end users don&#x27;t notice or care? In such a scenario brand would matter more than the best tech, and OpenAI is way out in front in brand recognition.

In organizations, everyone&#x27;s existence and position is politically supported by their internal peers around their level. Even google&#x27;s &amp; microsoft&#x27;s current CEOs are supported by their group of co-executives and other key players. The fact that both have agreeable personalities is not a mistake! They both need to keep that balance to stay in power, and that means not destroying or disrupting your peer&#x27;s current positions. Everything is effectively decided by informal committee.Founders are special, because they are not beholden to this social support network to stay in power and founders have a mythos that socially supports their actions beyond their pure power position. The only others they are beholden too are their co-founders, and in some cases major investor groups. This gives them the ability to disregard this social balance because they are not dependent on it to stay on power. Their power source is external to the organization, while everyone else is internal to it.This gives them a very special &quot;do something&quot; ability that nobody else has. It can lead to failures (zuck &amp; occulus, snapchat spectacles) or successes (steve jobs, gemini AI), but either way, it allows them to actually &quot;do something&quot;.

If he&#x27;s having an impact it&#x27;s because he can break through the bureaucracy. He&#x27;s not trying to protect a fiefdom.

I would say it more goes back to the Google Brain + DeepMind merger, creating Google DeepMind headed by Demis Hassabis.The merger happened in April 2023.Gemini 1.0 was released in Dec 2023, and the progress since then has been rapid and impressive.

Is there anything pointing to Brin having anything to do with Google’s turnaround in AI? I hear a lot of people saying this, but no one explaining why they do

Check the size and budget of Google iniatives. It’s unlimited

The Ghibli moment was an influencer fad not real advancement.

That&#x27;s a quite sensationalized view.Ghibli moment was only about half a year ago. At that moment, OpenAI was so far ahead in terms of image editing. Now it&#x27;s behind for a few months and &quot;it can&#x27;t be reversed&quot;?

Ouch.Maybe we&#x27;ll get some awesome FOSS tech out of its ashes?

&gt; I start to believe OAI is very much behindKara Swisher recently compared OpenAI to Netscape.

GPT 5.2 is actually getting me better outputs than Opus 4.5 on very complex reviews (on high, I never use less) - but the speed makes Opus the default for 95% of use cases.

i think the most important part of google vs openai is slowing usage of consumer LLMs. people focus on gemini&#x27;s growth, but overall LLM MAUs and time spent is stabilizing. in aggregate it looks like a complete s-curve. you can kind of see it in the table in the link below but more obvious when you have the sensortower data for both MAUs and time spent.the reason this matters is slowing velocity raises the risk of featurization, which undermines LLMs as a category in consumer. cost efficiency of the flash models reinforces this as google can embed LLM functionality into search (noting search-like is probably 50% of chatgpt usage per their july user study). i think model capability was saturated for the average consumer use case months ago, if not longer, so distribution is really what matters, and search dwarfs LLMs in this respect.<a href="https:&#x2F;&#x2F;techcrunch.com&#x2F;2025&#x2F;12&#x2F;05&#x2F;chatgpts-user-growth-has-slowed-report-finds&#x2F;" rel="nofollow">https:&#x2F;&#x2F;techcrunch.com&#x2F;2025&#x2F;12&#x2F;05&#x2F;chatgpts-user-growth-has-s...</a>

Add This to Gemini distribution which is being adcertised by Google in all of their products, and average Joe will pick the sneakers at the shelf near the checkout rather than healthier option in the back

OAI&#x27;s latest image model outperforms Google&#x27;s in LMArena in both image generation and image editing. So even though some people may prefer nano banana pro in their own anecdotal tests, the average person prefers GPT image 1.5 in blind evaluations.<a href="https:&#x2F;&#x2F;lmarena.ai&#x2F;leaderboard&#x2F;text-to-image" rel="nofollow">https:&#x2F;&#x2F;lmarena.ai&#x2F;leaderboard&#x2F;text-to-image</a><a href="https:&#x2F;&#x2F;lmarena.ai&#x2F;leaderboard&#x2F;image-edit" rel="nofollow">https:&#x2F;&#x2F;lmarena.ai&#x2F;leaderboard&#x2F;image-edit</a>

the trend I&#x27;ve seen is that none of these companies are behind in concept and theory, they are just spending longer intervals baking a more superior foundational modelso they get lapped a few times and then drop a fantastic new model out of nowherethe same is going to happen to Google again, Anthropic again, OpenAI again, Meta again, etcthey&#x27;re all shuffling the same talent around, its California, that&#x27;s how it goes, the companies have the same institutional knowledge - at least regarding their consumer facing options

Not trained on pro, distilled from it.

&gt; for benchmaxxing.Out of all the big4 labs, google is the last I&#x27;d suspect of benchmaxxing. Their models have generally underbenched and overdelivered in real world tasks, for me, ever since 2.5 pro came out.

This is obviously trained on Pro 3 outputs for benchmaxxing.

At this point in time I start to believe OAI is very much behind on the models race and it can&#x27;t be reversedImage model they have released is much worse than nano banana pro, ghibli moment did not happenTheir GPT 5.2 is obviously overfit on benchmarks as a consensus of many developers and friends I know. So Opus 4.5 is staying on top when it comes to codingThe weight of the ads money from google and general direction + founder sense of Brin brought the google massive giant back to life.
None of my companies workflow run on OAI GPT right now. Even though we love their agent SDK, after claude agent SDK it feels like peanuts.

Docs: <a href="https:&#x2F;&#x2F;ai.google.dev&#x2F;gemini-api&#x2F;docs&#x2F;gemini-3" rel="nofollow">https:&#x2F;&#x2F;ai.google.dev&#x2F;gemini-api&#x2F;docs&#x2F;gemini-3</a>Developer Blog: <a href="https:&#x2F;&#x2F;blog.google&#x2F;technology&#x2F;developers&#x2F;build-with-gemini-3-flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.google&#x2F;technology&#x2F;developers&#x2F;build-with-gemini-...</a>Model Card [pdf]: <a href="https:&#x2F;&#x2F;deepmind.google&#x2F;models&#x2F;model-cards&#x2F;gemini-3-flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;deepmind.google&#x2F;models&#x2F;model-cards&#x2F;gemini-3-flash&#x2F;</a>Gemini 3 Flash in Search AI mode: <a href="https:&#x2F;&#x2F;blog.google&#x2F;products&#x2F;search&#x2F;google-ai-mode-update-gemini-3-flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.google&#x2F;products&#x2F;search&#x2F;google-ai-mode-update-ge...</a>Deepmind Page: <a href="https:&#x2F;&#x2F;deepmind.google&#x2F;models&#x2F;gemini&#x2F;flash&#x2F;" rel="nofollow">https:&#x2F;&#x2F;deepmind.google&#x2F;models&#x2F;gemini&#x2F;flash&#x2F;</a>