It is technically either.
But the word form used in the US vernacular is to "wrack" one's brain (or brains). The word wrack is a variant of "wreck", to ruin or destroy.
This causes the dichotomy in "nerve-racking" and "nerve-wracking" : the US use tends toward the wracking as in wrecking, but racking can apply to torture as in stretching on a rack.
(see related question)
A homophone for "rack" is "wrack."
Usually rack one's brain. However, according to Oxford dictionary online, it's either:PHRASES rack (or wrack) one's brains make a great mental effort. -ORIGIN from Dutch rec, Low German rek 'horizontal bar or shelf', probably from recken 'to stretch, reach'.(also suggested is that the Dutch rec (framework) leads to the Middle English rakke)
Wrack My Brain was created in 1981.
The homonym for "rack" is "wrack." It is pronounced the same but has a different spelling and meaning.
Speaking of destruction we say wrack and ruin. Speaking of steering we say rack and pinion.
The adjective is spelled "nerve-wracking" (but also seen as nerve-racking).It is either from the variant wrack meaning to wreck, or rack as in the torture rack, which is seen in the idiom to rack one's brain (to stretch or exert).
Wrack, as in "sugar wrack" refers to marine vegetation, specifically seaweed or more specifically, kelp. Wrack is also a term used to refer to a thin cloud. https://en.wikipedia.org/wiki/Wrack
an island wrack is me and my best friend playing on the swings
Darren Wrack was born on 1976-05-05.
Matt Wrack was born on 1962-05-23.
Das geheimnisvolle Wrack was created in 1954.
The duration of Das geheimnisvolle Wrack is 1.25 hours.