The required compute seems a bit high: "We trained all CIFAR-10 models on 1xB200 GPU, and all ImageNet 64×64 models on 8xB200 GPUs. The largest CIFAR-10 model uses 20 B200 hours to train, and the largest ImageNet 64×64 model uses 640 B200 hours"
When I first learned about computer science at the age of 11 or so (and in 1982 or so) the first page of the text book put digital and analogue computers on what seemed to be an equal footing. And then proceeded to ignore the latter for the rest of the book. Apart from a few notable exceptions ( https://en.wikipedia.org/wiki/Phillips_Machine ) I've often wondered about analogue computing.
If you want to understand the issue with analog computers, design a SHA-256 circuit for one of them and consider the consequences of trying to push a megabyte of data through it. While that is an extreme example I chose precisely to make the issues clear, much real computing has many of the same characteristics, just distributed a bit more widely in time and space.
Or, to put it another way, you can make anything sound good if you consider only the positives and anything sound bad if you only consider the negatives. Analog computing sounds amazing when you read the brochure and consider only the positives. But when you bring the negatives back in, it makes sense why it is not frequently used. It is not a case of the mainstream keeping some great idea down because, uh, Big Digital or something, it's a case where digital computing turns out to be a stonking good idea and it's hard for the analog world to compete and it's virtually impossible for them to ever be anything but a niche.
Neural networks are an interesting possibility for a future successful niche, although even so, it would be neural networks specifically that may grow in importance and not analog computing in general. And I still wouldn't guarantee it'll be a good idea... we may have a lot of trouble keeping what would be very deeply nested analog circuitry stable in the real world and digital may still win out, e.g., an analog neural net that has a noticeable personality shift when it gets warmer may not be the best engineering solution. That's a question for 20 or 30 years from now.
And a general lack of reconfigurability to solve general problems. There’s been interest in analog neural networks for a long time.
Those problems you mention are important in music synthesis where people could live with limited reconfigurability but reliability is at a premium: synth players in early touring bands (e.g. Yes) had to be electronics technicians and instruments have to survive being packed in boxes and transported everywhere. The Yamaha DX-7 made FM synthesis mainstream because digital FM synthesis was absolutely reliable.
Quantum annealers (D-Wave machines) are basically analogue computers, with Josephson junctions as the primary component as opposed to oscillators. I wonder if they could render these images, too?
At the end of my undergrad, I remember a UW professor being poached by intel to work on analogue computing research project, the chair of the department at the time said that it was an opportunity that might not ever happen again and he had to take. I don’t think it went anywhere (since I never heard of intel coming out with a product), but I at least knew there was an attempt.
I love this ! Used to work at Rain AI on training neural networks in unconventional hardware - people often that computers don't necessarily have to be electronic digital - there is a whole domain dedicated to creating machines that can apply certain mathematical operations faster or more efficiently than their electronic counter parts. I created this site to try create a classification of that space:
Really interesting - if I understood the article correctly, they're simulating this on conventional hardware, so in order to get the proposed benefits, it would need to be implemented in some other electronic medium.
Very cool. I’m reminded of Wolfram’s pitch that neural nets are a search through the very broad computational complexity of the function space they describe; he did a little work to show that you could find similar behavior in other function spaces. These oscillators are yet again a different function space, and its cool they can be harnessed in this way.
The question of what physical / electronic phenomena is the most efficient yet large enough function space to be used for inference is a really good one to think about. I have no suggestions.
It’s not clear to me how this would ever be practical since it seems dependent on n^2 scaling.
You’ve got to wonder when you have an image generation demo why would you possibly have 64 x 64 pixel output as your demo?
If I’m understanding this properly to generate a 4K image, you need like 5 trillion point to point connections on the chip. Even if power use from the oscillators is zero that’s going to be an issue.
Yes I too am perplexed. I'm into audio synthesis so I feel I have somewhat better-than-average knowledge of oscillators, from the component or elementary mathematical level (depending on whether they're analog or digital) to complex interactions for fun and profit (frequency, phase, ring modulation).
These are cool results but I was disappointed not to find any discussion of where oscillator array technology stands today what the manufacturing challenges/opportunities might be. It seems like it would be prohibitively expensive for anything beyond minimal networks of a few hundred nodes that could be used in sensors. Even if you have perfectly consistent oscillators that synchronize to each other within very fine tolerances, wiring them up to each other is still a massive headache.
What they are trying to achieve is to demonstrate that the coupling approach works in a simulated physics environment (O(n^2) as you point out) so that they can then build CMOS circuits that create actual oscillators and then let the laws of physics do the computation. This is a very bold vision!
And anyone who has done an introductory course in VLSI design would know that capacitance (coupling) is something you usually want to get rid of. However, all kinds of amazing analog circuits have been developed over the decades that exploit coupling effects. So, their idea is not outlandish at all.
Which idea is not outlandish? Physical computing? I agree physical computing is a fascinating topic.
But specifically what they’ve simulated here? I don’t see how that would ever work in real life scaled up to any kind of real size.
I’m not criticizing them for starting out small. Lots of things can be proven with small models. I’m saying in principle, I don’t see how this will work unless there’s some fundamentally new technique that is currently not known about. Maybe they have some secret idea but they haven’t shown it here.
At 5k to 10k nodes aren't they in the ballpark of a single layer from a scaled up conventional model? Rather than scaling further presumably you could stack these. However for a physical implementation ~100M interconnects seems questionable to me (but I know next to nothing about hardware engineering TBF) so I wonder if they intend to move to a partially connected model similar to the gyroscopic model of computation that the article links to.
I read through the article, and I'm not sure this is dependent on quadratic scaling.
Are they allowing all oscillators to influence all others, or are they picking modalities where the influences can be limited to some maximal fixed degree?
One would imagine that there'd be a variety of different topologies available to explore. Even if during training the treatment was fully connected, one could imagine the training itself biasing towards a maximal fixed degree per oscillator, and then inference later operating on a quantized version of that that drops the low-weight influences to zero.
Do you mean that they may get away with less oscillators because of the decoder layer? Well there’s the rub isn’t it, the more work you have done by a software layer the less power you’ve proportionally saved by having it be done by physical computing.
But let’s spitball here what would you estimate would be needed in number of oscillators and interconnects for a 4K image?
Conventional image generators still have to process n^2 connections so I don't think that observation is a valid objection in and of itself.
One thing I'm unclear on is that their total parameter count scales similarly to conventional models but many of those conventional models incorporate convolutions. I wonder how interconnect count (as opposed to unique parameters) compares to performance?
As to 4k images, I'm not clear how much farther their current architecture would be expected to scale. Single layer networks aren't parameter efficient compared to deep networks; I'd naively assume that to also apply here. That said given their results so far with what amounts to a single layer the naive assumption begins to seem questionable.
Think of the models making progress on CIFAR-10, ImageNet, CelebA, etc. 15 years ago. They had issues too and weren't just scaled-up as is to the architectures we have today.
This method is cool and the post explains it well. It would, however, be good to get more detail on the energy efficiency they flag as their motivation: is this model actually more energy efficient than the comparators they highlight?
It seems like total parameter count is more or less on par with conventional approaches so any gains won't be from there.
We can implement coupled oscillators in hardware but are the couplings and frequencies programmable? If they're being streamed in I guess you'd still have a memory bandwidth bottleneck and associated energy usage. If not then the fair comparison is to a conventional model hardcoded in an ASIC which AFAIU is actually quite energy efficient.
Do the parameters in these harmonic systems compress better? Instead of needing to hold individual parameters for each oscillator, could groupings of oscillators be instead be described with its output over a given time and then just reverse that output to get the original parameters (I’m thinking the output is like an FFT of the oscillators which is a single value, then do an inverse FFT to get the original oscillator parameters etc)
Very cool work - refreshing to see a of different approach. I learned about Kuramoto oscillators many years ago from a book called Sync, by Steven Strogatz, which I highly recommend.
(Disclaimer, not my area of expertise.) It appears to be adjacent but more general. There's an entire collection of methods (including reservoir computing) that conceptually resemble or are based on physical systems in one way or another. This appears to be an attempt to develop a new method that natively takes place as a physical process that we could readily implement in hardware.
Yes, I had the same question. I don’t think so, as currently designed. It trains to specific points / classes in an embedding space. They didn’t discuss how one might go to non-trained points in the paper as far as I could read, and they did show some visualization around the idea that the runs aim at / around set points in the space.
The required compute seems a bit high: "We trained all CIFAR-10 models on 1xB200 GPU, and all ImageNet 64×64 models on 8xB200 GPUs. The largest CIFAR-10 model uses 20 B200 hours to train, and the largest ImageNet 64×64 model uses 640 B200 hours"
20 B200 hours for CIFAR-10 seems like a lot...
When I first learned about computer science at the age of 11 or so (and in 1982 or so) the first page of the text book put digital and analogue computers on what seemed to be an equal footing. And then proceeded to ignore the latter for the rest of the book. Apart from a few notable exceptions ( https://en.wikipedia.org/wiki/Phillips_Machine ) I've often wondered about analogue computing.
If you want to understand the issue with analog computers, design a SHA-256 circuit for one of them and consider the consequences of trying to push a megabyte of data through it. While that is an extreme example I chose precisely to make the issues clear, much real computing has many of the same characteristics, just distributed a bit more widely in time and space.
Or, to put it another way, you can make anything sound good if you consider only the positives and anything sound bad if you only consider the negatives. Analog computing sounds amazing when you read the brochure and consider only the positives. But when you bring the negatives back in, it makes sense why it is not frequently used. It is not a case of the mainstream keeping some great idea down because, uh, Big Digital or something, it's a case where digital computing turns out to be a stonking good idea and it's hard for the analog world to compete and it's virtually impossible for them to ever be anything but a niche.
Neural networks are an interesting possibility for a future successful niche, although even so, it would be neural networks specifically that may grow in importance and not analog computing in general. And I still wouldn't guarantee it'll be a good idea... we may have a lot of trouble keeping what would be very deeply nested analog circuitry stable in the real world and digital may still win out, e.g., an analog neural net that has a noticeable personality shift when it gets warmer may not be the best engineering solution. That's a question for 20 or 30 years from now.
Is SHA-256 something that makes sense in analog realm, or is it something that only needs to exist due to digital constraints?
Noise and component imprecision has always limited analog computing.
In neutral networks, we seem to be pushing towards ever lower precision floats, and we use noise for all sorts of useful things.
And a general lack of reconfigurability to solve general problems. There’s been interest in analog neural networks for a long time.
Those problems you mention are important in music synthesis where people could live with limited reconfigurability but reliability is at a premium: synth players in early touring bands (e.g. Yes) had to be electronics technicians and instruments have to survive being packed in boxes and transported everywhere. The Yamaha DX-7 made FM synthesis mainstream because digital FM synthesis was absolutely reliable.
Analog synths are a lot more reliable these days though.
Also true: all computing is analog computing.
My father designed processors. He says all electronics are analog. Some just pretends to act digital.
Quantum annealers (D-Wave machines) are basically analogue computers, with Josephson junctions as the primary component as opposed to oscillators. I wonder if they could render these images, too?
At the end of my undergrad, I remember a UW professor being poached by intel to work on analogue computing research project, the chair of the department at the time said that it was an opportunity that might not ever happen again and he had to take. I don’t think it went anywhere (since I never heard of intel coming out with a product), but I at least knew there was an attempt.
Take a look at extropic. AFAICT it's a form of analog computer.
I love this ! Used to work at Rain AI on training neural networks in unconventional hardware - people often that computers don't necessarily have to be electronic digital - there is a whole domain dedicated to creating machines that can apply certain mathematical operations faster or more efficiently than their electronic counter parts. I created this site to try create a classification of that space:
https://computers.tugdual.fr/
Really interesting - if I understood the article correctly, they're simulating this on conventional hardware, so in order to get the proposed benefits, it would need to be implemented in some other electronic medium.
Very cool. I’m reminded of Wolfram’s pitch that neural nets are a search through the very broad computational complexity of the function space they describe; he did a little work to show that you could find similar behavior in other function spaces. These oscillators are yet again a different function space, and its cool they can be harnessed in this way.
The question of what physical / electronic phenomena is the most efficient yet large enough function space to be used for inference is a really good one to think about. I have no suggestions.
It’s not clear to me how this would ever be practical since it seems dependent on n^2 scaling.
You’ve got to wonder when you have an image generation demo why would you possibly have 64 x 64 pixel output as your demo?
If I’m understanding this properly to generate a 4K image, you need like 5 trillion point to point connections on the chip. Even if power use from the oscillators is zero that’s going to be an issue.
Yes I too am perplexed. I'm into audio synthesis so I feel I have somewhat better-than-average knowledge of oscillators, from the component or elementary mathematical level (depending on whether they're analog or digital) to complex interactions for fun and profit (frequency, phase, ring modulation).
These are cool results but I was disappointed not to find any discussion of where oscillator array technology stands today what the manufacturing challenges/opportunities might be. It seems like it would be prohibitively expensive for anything beyond minimal networks of a few hundred nodes that could be used in sensors. Even if you have perfectly consistent oscillators that synchronize to each other within very fine tolerances, wiring them up to each other is still a massive headache.
I bet 5 million coupled oscillators, all slightly detuned, would sound freakin' amazing.
Here's 1000: https://www.youtube.com/watch?v=c3wk9WWTfNs&t=338s
What they are trying to achieve is to demonstrate that the coupling approach works in a simulated physics environment (O(n^2) as you point out) so that they can then build CMOS circuits that create actual oscillators and then let the laws of physics do the computation. This is a very bold vision!
And anyone who has done an introductory course in VLSI design would know that capacitance (coupling) is something you usually want to get rid of. However, all kinds of amazing analog circuits have been developed over the decades that exploit coupling effects. So, their idea is not outlandish at all.
But wouldn't capacitance as it naturally occurs be only to immediate neighbors? Not n^2 as in their model.
Which idea is not outlandish? Physical computing? I agree physical computing is a fascinating topic.
But specifically what they’ve simulated here? I don’t see how that would ever work in real life scaled up to any kind of real size.
I’m not criticizing them for starting out small. Lots of things can be proven with small models. I’m saying in principle, I don’t see how this will work unless there’s some fundamentally new technique that is currently not known about. Maybe they have some secret idea but they haven’t shown it here.
At 5k to 10k nodes aren't they in the ballpark of a single layer from a scaled up conventional model? Rather than scaling further presumably you could stack these. However for a physical implementation ~100M interconnects seems questionable to me (but I know next to nothing about hardware engineering TBF) so I wonder if they intend to move to a partially connected model similar to the gyroscopic model of computation that the article links to.
Doesn't that require quadratically-many wires to connect all the processing units?
I read through the article, and I'm not sure this is dependent on quadratic scaling.
Are they allowing all oscillators to influence all others, or are they picking modalities where the influences can be limited to some maximal fixed degree?
One would imagine that there'd be a variety of different topologies available to explore. Even if during training the treatment was fully connected, one could imagine the training itself biasing towards a maximal fixed degree per oscillator, and then inference later operating on a quantized version of that that drops the low-weight influences to zero.
The oscillating elements don't map directly to pixels. Conventional models also have n^2 parameters.
Well image generators work differently…
Do you mean that they may get away with less oscillators because of the decoder layer? Well there’s the rub isn’t it, the more work you have done by a software layer the less power you’ve proportionally saved by having it be done by physical computing.
But let’s spitball here what would you estimate would be needed in number of oscillators and interconnects for a 4K image?
Conventional image generators still have to process n^2 connections so I don't think that observation is a valid objection in and of itself.
One thing I'm unclear on is that their total parameter count scales similarly to conventional models but many of those conventional models incorporate convolutions. I wonder how interconnect count (as opposed to unique parameters) compares to performance?
As to 4k images, I'm not clear how much farther their current architecture would be expected to scale. Single layer networks aren't parameter efficient compared to deep networks; I'd naively assume that to also apply here. That said given their results so far with what amounts to a single layer the naive assumption begins to seem questionable.
Think of the models making progress on CIFAR-10, ImageNet, CelebA, etc. 15 years ago. They had issues too and weren't just scaled-up as is to the architectures we have today.
This method is cool and the post explains it well. It would, however, be good to get more detail on the energy efficiency they flag as their motivation: is this model actually more energy efficient than the comparators they highlight?
It seems like total parameter count is more or less on par with conventional approaches so any gains won't be from there.
We can implement coupled oscillators in hardware but are the couplings and frequencies programmable? If they're being streamed in I guess you'd still have a memory bandwidth bottleneck and associated energy usage. If not then the fair comparison is to a conventional model hardcoded in an ASIC which AFAIU is actually quite energy efficient.
Do the parameters in these harmonic systems compress better? Instead of needing to hold individual parameters for each oscillator, could groupings of oscillators be instead be described with its output over a given time and then just reverse that output to get the original parameters (I’m thinking the output is like an FFT of the oscillators which is a single value, then do an inverse FFT to get the original oscillator parameters etc)
Very cool work - refreshing to see a of different approach. I learned about Kuramoto oscillators many years ago from a book called Sync, by Steven Strogatz, which I highly recommend.
Readers care, this requires a nice amount of physics knowledge to really understand. Not too advanced but still, physics.
> However, the trade-off with our approach is that it requires a more complex loss that operates given only generated samples.
This kind of reminds me of DCT in lossy image compression, but in reverse.
Not at all related but still reminds me a bit of FM synthesis
Is this somewhat related to reservoir computing?
(Disclaimer, not my area of expertise.) It appears to be adjacent but more general. There's an entire collection of methods (including reservoir computing) that conceptually resemble or are based on physical systems in one way or another. This appears to be an attempt to develop a new method that natively takes place as a physical process that we could readily implement in hardware.
Can this even make an image having more than one "class"? Can it make an image of an astronaut riding a horse on the moon?
Yes, I had the same question. I don’t think so, as currently designed. It trains to specific points / classes in an embedding space. They didn’t discuss how one might go to non-trained points in the paper as far as I could read, and they did show some visualization around the idea that the runs aim at / around set points in the space.
finally, a way to generate images that's slower AND worse. progress.
I didn't really understand anything...lgtm