Indeed!
You have not idea how confusing it is for me with some of those. By the way. We have an idiom very similar in meaning to "pulling my leg" but it would translate as "pulling the hair". It is a little more complicated, because if you actually pull someone by grabbing their hair, that would be "tirar del pelo", but the idiom for when someone is pranking you is "tomar el pelo", which is...softer. Like taking someone hair in your hand but not actually pulling it. But anyways, it is about hair and not legs.
Oh, I'm quite aware that other languages have their odd/absurd idioms, too!
So no, I'm not surprised you find English difficult in this way... as with many things, English is just a bit worse in this regard than most languages are.
You may have heard: other languages have "loanwords." English drags the other language down an alleyway, hits it over the head, and goes through the pockets for loose vocabulary and grammar.