In his first post-Repentance Lore Stream, Edmund McMillen revealed that The Belt is inspired by a real belt that his step-father would crack menacingly to threaten him with.
The Belt granting a speed up is a nod to the act of running faster out of fear of being hit.
In this way, The Belt is a counterpart to Wooden Spoon, which is also a speed-up item based on a specific object used by McMillen's parents to abuse him.