60 Watts appears to be the limit of test fixture under load
In my experience, that's about the limit for efficient operation of a non-synchronous converter. Much higher than that and losses start to really add up in the diode.
Synchronous converters are a bit more of a challenge, but can provide more power at the output.
You are correct in that the test layout is also a limiting factor. You have parasitic everythings in there - Inductance, capacitance, you name it. It all adds up and works against you.
I can't emphasize enough that everything in the "grunt" portion of the circuit needs to be as close together as physically possible. A 3 inch wire here, 4 inch there, doesn't seem like much, and for DC it isn't. But at the switching frequencies that converters typically run at, they become inductors and cause switching pulses to "soften" and all kinds of other havoc. In fact, get enough inductance in the wrong area, and it will leave you scratching your head wondering why you have a pile of popped MOSFETs.
By close together I mean 1/4 inch. And heavy. At the power levels you're after, you have to minimize the losses everywhere you possibly can, or you're working backwards against yourself.
Your layout reminds me of my first few attempts at making a converter as well, and they all failed, miserably. Once I understood the importance of physical proximity and layout, the magic smoke stayed in and the power started flowing.
For learning purposes, there is inevitably a certain amount of rat nest in the mix. The sooner you can get away from it though, the better. For an idea of how it's typically done, examine a manufactured buck converter. You will notice that the 3 power handling components (tranny, inductor, diode) are very close together. VERY close. And the caps are not far from them either. Which brings me back to caps.
And caps. Caps, caps, and more caps. Caps on the input. Caps on the output. Low ESR caps. Right next to the switching components. Lots and lots of caps. It may seem like a couple 4700uF elecrolytics would have more brass than a dozen or so of their low ESR 47uF brethren, but at those frequencies, the inductance in the leads and plates of the 4700s become a significant issue, and reduce (if not just about eliminate) their effectiveness. Larger caps are ok to have in the mix, in fact I recommend that you keep them there. Just move them "further out" away from the switching components and use the smaller, more solid counterparts up close and personal with the switching components. You'd be amazed at how much this will improve performance. I usually use a few different values too - my theory there is it "detunes" the rails, and helps suppress and absorb ringing and pulses across a wider spectrum. Might be overkill, but I do it anyway.
It's not so much the actual switching frequency either, even though this of course plays a significant role. The switching frequency directly affects losses because the higher it is, the more often the transistor is in "linear" mode, which is essentially resistive, which translates to heat. But that's only part of the equation. The
slew rate (time it takes the transistor to transition from fully off to fully on and vice versa) is
also a significant contributor to the amount of power lost as heat in the transistor. The more time it takes to transition, the longer the transistor stays in the linear region on each cycle, which translates to even more heat.
Buck converters are finnicky little bastards, and it doesn't take much for the losses to add up and make them much less effective than they could be.
And keep things close together! (Think I already mentioned that!)
Steve