I guess bacteria is the problem, multiplying while you're away!
You have food particles and other germs and bacteria still in your mouth when you drink out of the bottle. So everytime you go to take another swig, not only are you adding more nastiness to what's already in the bottle, But your also drinking all the other stuff that's in the backwash.
I never wash my coffee cup.
Germs from back flush, and mold... eventually those thing will start a mini laboratory inside your bottle.
not much. unless you backwash your filthy germs into it a lot.
Because germs are more hardy than you think.