Your voltage regulator is placed at the positive lead because that is the convention used to design the components - nothing too mystical about it. It is the positive voltage potential that is being regulated. The negative terminal is acting as the reference point (common node) in the circuit.
Depending on the voltage regulator in use, we can get a regulated positive or negative voltage, at whichever voltage we want. The LM78XX voltage regulators are a popular kind for regulating and outputting positive voltage, while the LM79XX are a popular series of regulators for negative voltage.
The positive side of a battery is where the electrical current flows out, while the negative side is where the current flows in. These sides are commonly referred to as the positive and negative terminals respectively. How can I identify the positive and negative terminals of a battery?
Poles: In a battery, the negative side is commonly referred to as the cathode or the negative pole. It is the end of the battery where electrical current flows out. The negative pole is often the larger terminal and can be identified by its negative symbol or a minus (-) sign.
If electrons make one side of the battery negative, then the other side is lacking those electrons and wants them. Because the positive terminal is lacking those electrons it has a much more positive voltage. It likely has a lot more protons (which are positive) than the negative side of the battery.
If the other end of the resistor is connected to the positive pole of the battery, the extra electrons will want to travel from the resistor to the positive pole of the battery following the charge density gradient. Now the chemical process within the battery is "triggered" and these electrons are again "moved" to the negative pole of the battery.