How do you remove rust from stainless steel water bottles? Posted on September 22, 2022 By admin Put a teaspoon amount of baking soda into the bottle. Fill the rest up with water. Scrub the inside of the lid with the baking soda solution too. Let the bottle sit overnight and then rinse it out with water the next morning.