Glance at the examples of what the "taken out" looks like for the various stress tracks you're considering. Then see which applies to what your NPC is doing most closely. Technically it could be any of the three.
Social is more of a James Bond style torture.
"No Mr. Bond, I expect you to die." or maybe a generic
"After I'm done with you I'm going to start on the girl..."Physical is pretty obvious. Waterboarding, whipping, attacking various non-vital body parts, etc, are all physical. The thing to keep in mind here is that the physical stress is incidental. Your bad guy doesn't have to take the character out physically. He's already contained somehow. In fact, taking the character out physically is an inconvenience to the NPC because then the torture has to pause while your PC is brought around again.
Mental is the usual goal here. Your NPC wants the character to break, to be unable to consider resisting what he wants. The thing is, the NPC can't directly attack your PCs mind so he has to go the indirect route. But ultimately what he's going for is a mental consequence that he can tag to pressure your PC into giving up whatever it is the NPC wants.
Just FYI, this is all opinion. I have no actual knowledge of torture in the real world, this is just how it seems to be used in stories. Also, not all NPCs who are willing to torture someone know how to go about it efficiently. Some will just beat the tar out of the good guys, knock them out, then be pissed that they haven't gotten what they wanted yet. And even the best make mistakes. So consider what your NPC knows about the subject, what he wants, and (most importantly) what would do your story the most good as you make this happen. PCs that are powerless for any length of time can quickly lead to players that aren't having fun. Keep a close eye on the mood of your group.