Blowing up a balloon involves forcing additional air particles from your lungs into the balloon. These particles hit the inside walls of the balloon creating enough pressure to force the rubber of the balloon to expand and the balloon to inflate. The collision of these particles with the walls creates a high-pressure environment inside the balloon relative to the atmospheric pressure around it. This is why when a balloon is released, the high-pressure air flows out of the balloon to the low-pressure air surrounding it. "Winds blow from high to low."
When the balloon is placed inside the bottle, it will not inflate, since the bottle is already filled with air particles with no escape route. This is a great demonstration that air takes up space. The air inside the bottle compresses a little bit but not enough to permit the balloon to inflate.
However, when you punch a hole in the bottle, the air molecules in the bottle have an exit. They are pushed out as the balloon fills the space inside, resulting in room for the balloon to inflate.
If the hole is then plugged, the balloon stays inflated even when the mouth is removed. This is because the high pressure air in the balloon pushes outward harder than the low pressure air in the bottle. The air in the balloon pushes out against the walls, keeping it inflated. When the hole is unplugged, air flows back into the bottle. The air pressure in the bottle increases and collapses the balloon.