Softmax chip

CSE 122 (VLSI) · Tiny Tapeout · Sky130 · Verilog · cocotb · March–October 2026

A softmax chip with no lookup tables, built to fit in the smallest Tiny Tapeout tile. The first version didn’t fit. The third one finally proved the idea worked.

Source on GitHub →

v3 placed and routed in a 161×112 µm Tiny Tapeout tile: about 1,200 Sky130 cells, metal 1–4 routing and gold met4 power straps.

Picking a project

For CSE 122 we had to design something for Tiny Tapeout: a few hundred gates, done in about a week. I wanted a project I could actually talk about, and since I was applying to NPU jobs, something AI-related. A clock-gated multiply-accumulate unit was the safe option, but there’s not much to say about a MAC.

Softmax was more interesting. It runs in every attention layer, it needs e^x for every input, and hardware usually gets e^x from a lookup table. My idea was to skip the table and build the curve out of a few straight lines instead, and see if that came out smaller.

Version 1: built for throughput

My first design was built for throughput: four exponential units running in parallel, a Newton–Raphson reciprocal for the sum, and four multipliers to scale the outputs. That’s what the accelerators I’d been reading about used, so that’s what I used. Honestly, I didn’t understand those parts well enough yet to know whether a chip this small needed them. I was also estimating the size like it was an FPGA, at around 220 LUTs. A real chip doesn’t have spare LUTs lying around, and every multiplier costs hundreds of gates.

The first full build came back at 8,881 gates, three times what the tile could hold, with about three hours left before the deadline. I changed the goal to something I could still hit: fit in the tile, and still be smaller than the lookup table it was replacing. I cut the bit widths, swapped the multipliers for shifts, and replaced the reciprocal and the scaling with two small lookup tables. That got it down to about 1,400 cells, and it passed GDS, precheck and the gate-level test.

Cell count at each version 1 commit, re-run from the Git history with the same Yosys version and command.

It passed, but I couldn’t really answer the obvious question: if it has two lookup tables in it, what was the point of a LUT-less softmax? When I came back to it months later, two things stood out:

I sped up the wrong part. The chip can only send out one byte per clock, so all that parallel hardware was waiting on the pins anyway. Those blocks were 58% of the original design.
The inputs were too simple. With whole-number inputs there are only seven different exponentials it can ever need, so a tiny table is already exact and an approximation has nothing to beat. The outputs were also way off: an average error of 0.91, where 0 is perfect and 2 is completely wrong.

The big lesson for me: use the fast version of something because you need it, not because it’s the one everyone uses.

Version 2: checking the premise

Before building it again, I wanted to know if the idea was even right. So I built the boring version: an exact seven-entry table, plus one divider instead of the reciprocal and four multipliers. It was smaller (about 6,500 against 11,200 µm²) and way more accurate (average error 0.002). For whole-number inputs, approximating just doesn’t help, so version 1 never had a chance.

Version 3: the right technique for the right inputs

Approximating only pays off when the exact answer is expensive, and that depends on how precise the inputs are. With fractional inputs, the table gets bigger with every extra bit, while the straight-line version barely grows:

Area of one exponential unit as the input step shrinks: exact table against shift-only piecewise-linear (PWL) curves.

So v3 takes 16-bit inputs in steps of 1/256, where an exact table would need 1,597 entries. The exponential is my original idea, done properly this time: eight straight-line segments, where every slope is just one or two bit shifts, so there are no multipliers. The worst error is 1.84 out of 255. And since the outputs leave one at a time anyway, there’s only one exponential unit and one divider, shared by all four inputs.

v3 against an exact-table twin built in the same frame and synthesised the same way.

The LUT-less exponential is less than half the size of the table, and the whole chip is about 20% smaller, with an average error of 0.006. On October 16, 2026, v3 passed the full Tiny Tapeout flow: GDS, precheck and the gate-level test. So the idea was right. I just needed the right inputs for it.

How the divider works

Funnily enough, the divider is just long division, the way you learned it in school, but in binary. Each clock it shifts the remainder over, subtracts if it can and writes a 1, or writes a 0 if it can’t. Eight clocks, eight bits, one subtractor. Here’s a real run from simulation:

Inputs 1.0, 2.0, 3.0 and 4.0: eight bytes in, four exponentials summed through the shared lane, then four divisions. The outputs 8, 22, 60 and 164 match true softmax exactly.

Verification

The cocotb tests run 27 hand-picked cases and 500 random ones against real softmax. Every block has its own test too, and the exponential is checked against every one of its 2,049 possible inputs. The full chip matches a bit-exact Python model on 3,000 runs, and the gate-level version built from real Sky130 cells gives the same answers.

Down at the cells

About 18×12 µm of the tile: metal 1 (violet), metal 2 (pink), metal 3 (teal) and a met4 power strap (gold).

The cells underneath with the metal hidden: red polysilicon gates over green diffusion, with blue local interconnect and contacts.

The layout images were generated for this page from the v3 netlist, placed and routed with a small custom placer and router and rendered in KLayout. The Tiny Tapeout build that passed used the official flow.