It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
Gaming used to be done in arcades for 25c a play*.
Over time NeoGeo realized they could sell you the hardware to play at home (āon-premā) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
āø»
* Thatās for video games. In fairground arcades youād literally pay for tokens then use tokens for play. The analogy almost writes itself.
If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. Theyāre getting there very fast.
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO.
Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but their power rack boxes are effectively a little mainframe (check the IO stats on those things) and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.
You can try open models via OpenRouter and omp without having to buy or rent any GPU. I routinely try open models to see where they are. GLM 5.3 with DSFV4.1 as advisor via omp feels Opus 4.6 level, i.e. totally able to ship stuff.
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but Iām ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I donāt pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies itās very freeing because itās like having a company of people working for you, except they are not human so theyāre not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you donāt have to worry about their feelings or that they wonāt do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me itās a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
In OMP just setting the "task" role (which controls the default subagent model choice) to some cheap Flash model like Deepseek or Gemini goes a long way towards making better use of your quota.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
"Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
I didn't think that IBM had gear that mortals would want to run. If you're not running a mainframe, what do they have to sell you? They still sell Power servers and it looks like they have an AI card that can be used on both. But, if we're talking about the IBM CEO talking about on-premises, what are they really talking about hardware-wise? It's not something they would sell, is it?
This is more of a statement for their consulting customers, right?
Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
Of course IBM - the guys that sell you mainframes - would be saying this.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
The guy that sells mainframes also gives away the Granite models for free. I've found them to be quite capable in content retrieval and long-running workflows, even (or especially?) the small ones.
Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.
āThe question becomes: how do you do that?ā he said. āThe question that the enterprise struggles with is: āWhere are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that youāre doing the right caching of results so you get an incredible response?āā
Has he heard of āagentic AIā? You simply fire up Claude or ChatGPT and ask it how to do all of the above.
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
What happens when your competitors invest in better and more expensive tools or materials in general? They (potentially) make better products, but those products also cost more. It's turtles all the way down.
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if itās true. And maybe it is true! But IBMās CEO saying itās true tells me just as much as if a frontier lab said itās not: nothing.
Iāve been in tech for a long timeā¦ I have no idea what IBM does. Now, I get that Iām not the customer, that their customer is ābusinessā but not my businessā¦ noā¦ bigger business. The biggest of businesses I assume. I think it must either be a club Iām not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because āitās IBM, it must be importantā.
Review the specifications for the latest Telum II chip if you need a hint. 5.5ghz base clock.
If you absolutely need to scale something vertically, IBM Z is the final stop on the line.
X86 is great at hosting an entire farm of applications. You would never want to install salesforce or gitlab on a mainframe. That would completely defeat its purpose. If you've never encountered an appropriate workload, IBM might never make intuitive sense.
Ironically one of the smaller places I've worked at tried to use an IBM product once - I'm still not clear on what it was meant to do or how it was better than any cheaper alternative, so I assume someone in leadership was fooled by a very impressive pitch.
Lots of work went into integration and it never went live.
OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)
It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
Gaming used to be done in arcades for 25c a play*.
Over time NeoGeo realized they could sell you the hardware to play at home (āon-premā) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
āø»
* Thatās for video games. In fairground arcades youād literally pay for tokens then use tokens for play. The analogy almost writes itself.
** The model, not the video game.
If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. Theyāre getting there very fast.
How much did you shell out for this setup? Whatās the TTFT and the output tokens/sec rate? How much electricity does it consume?
Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
At work two different departments have it on their radar to do - apparently there are GPUs waiting to be used for something like it.
Smart move to get them bought ahead of time.
8 gpus is cheap now?
Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
Depends on context of what people build.
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO. Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but their power rack boxes are effectively a little mainframe (check the IO stats on those things) and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.
You can try open models via OpenRouter and omp without having to buy or rent any GPU. I routinely try open models to see where they are. GLM 5.3 with DSFV4.1 as advisor via omp feels Opus 4.6 level, i.e. totally able to ship stuff.
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but Iām ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I donāt pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies itās very freeing because itās like having a company of people working for you, except they are not human so theyāre not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you donāt have to worry about their feelings or that they wonāt do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me itās a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
In OMP just setting the "task" role (which controls the default subagent model choice) to some cheap Flash model like Deepseek or Gemini goes a long way towards making better use of your quota.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
But hardware to run them is conveniently something IBM would be happy to sell you.
If IBM would ship a piece of affordable hardware that can run large LLMs efficiently I'd be lining up.
"Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
I didn't think that IBM had gear that mortals would want to run. If you're not running a mainframe, what do they have to sell you? They still sell Power servers and it looks like they have an AI card that can be used on both. But, if we're talking about the IBM CEO talking about on-premises, what are they really talking about hardware-wise? It's not something they would sell, is it?
This is more of a statement for their consulting customers, right?
Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
I think they will need a far more aggressive management before such a thing could happen. But I can dream about it.
> It doesn't make sense that IBM is just watching NVIDIA take its lunch.
If you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
Not till Jensen buys his next leather jacket.
Of course IBM - the guys that sell you mainframes - would be saying this.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
The guy that sells mainframes also gives away the Granite models for free. I've found them to be quite capable in content retrieval and long-running workflows, even (or especially?) the small ones.
Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
RIP frontier AI, the top customers can easily self host
What happens when their competitors decide to pay for a more intelligent frontier model?
Cool! That's when you pay the difference in marketing and auto-win your sector! People don't actually buy the best, they buy what they think is.
What happens when your competitors invest in better and more expensive tools or materials in general? They (potentially) make better products, but those products also cost more. It's turtles all the way down.
You buy the competitor and kill their culture from the inside, ruin them and take a tax loss on the company.
depends on the economics and much more, the best Ai cannot do a lot of things, 90% of human tasks by some reports
we technologists tend to extend what we see and think to other realms where it doesn't make sense
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if itās true. And maybe it is true! But IBMās CEO saying itās true tells me just as much as if a frontier lab said itās not: nothing.
"CEO Said A Thing!" Journalism
https://karlbode.com/ceo-said-a-thing-journalism/
Great term. Plus the related "Politician Said a Thing!" journalism
thank you, I was trying to re-find this the other day
But you know - it would be funny if somehow Open Weights are the thing that completely save IBM
Linux already served as a life-preserver for them also.
Itās a conversation point off which a discussion can be had
the only credible "conversation" is "well he would say that, wouldn't he?"
"we are delighted..." said a spokesman
Iāve been in tech for a long timeā¦ I have no idea what IBM does. Now, I get that Iām not the customer, that their customer is ābusinessā but not my businessā¦ noā¦ bigger business. The biggest of businesses I assume. I think it must either be a club Iām not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because āitās IBM, it must be importantā.
Review the specifications for the latest Telum II chip if you need a hint. 5.5ghz base clock.
If you absolutely need to scale something vertically, IBM Z is the final stop on the line.
X86 is great at hosting an entire farm of applications. You would never want to install salesforce or gitlab on a mainframe. That would completely defeat its purpose. If you've never encountered an appropriate workload, IBM might never make intuitive sense.
Also, the roadmap:
https://newsroom.ibm.com/2026-08-24-ibm-unveils-next-generat...
Ironically one of the smaller places I've worked at tried to use an IBM product once - I'm still not clear on what it was meant to do or how it was better than any cheaper alternative, so I assume someone in leadership was fooled by a very impressive pitch.
Lots of work went into integration and it never went live.
> I have no idea what IBM does
For one thing, amongst many, IBM mainframes process 70% of global transactions on a value basis
Good enough for the old world, but I'm not so sure it's good enough for what's coming.
What do you think is coming that isn't already in the old world?
Cerebras with Claude 8?
OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)