I cannot emphasize enough the importance of cleanliness when dealing with baby goats, because of their lowered immune system. Baby goats have little to no immune system until they are about five weeks old, so they are extremely exposed and susceptible to all kinds of sickness. Kid goats need clean buckets, clean nipples, clean hosing/straw, and live in a clean environment.
So, clean, clean clean!
Keep them clean! As much as is possible, keep all feeding equipment as clean as is humanly possible to avoid bacteria buildup. Bottles should be used only once and then washed. Milk pails, hosing & nipples should be washed every day, especially during the summer months when the flies are worse. During the winter months it can be stretched to every second day. Notice how one of the straws / tubing going into the milk is a little dark; it probably needs to be washed.
I like to mix good old vinegar mixed with regular dish soap to clean all the feeding equipment. I use a bottle brush to clean the bottles out, and for the hosing on the tube feeders, I use boiling hot water and a hose brush (The Kids and Ewe is where I got mine).
Baby goats like to suck on just about everything! So make sure their boxes are clean. Some people use strong (and expensive) disinfectant cleaners, but we find it effective enough to use a mild dish detergent and boiling water. Another option is to buy cardboard boxes for this stage.
How often should you wash the boxes? Well, that depends on how many babies you have coming, and on how long you keep them in the box. Use your discretion, but don't go putting a brand new baby in a filthy box - that would be asking for trouble and pretty soon you will end up with sick (and probably soon dead) kids. Those wet, dirty places are just the environment that Coccidia and crypto thrive in.
Of course you won't be going into the goats' pens with a pail of soapy water every week (or will you?)... but be sure to keep the pens as clean and fresh as an animal pen can be. Add to their bedding as necessary.
Rules of thumb:
If they're getting dirty, bed them.
If the bedding pack is dark, bed them.
Be sure to go into the pens to wash their water bowls at least once a day. Baby goats like to play with anything and climb on everything - including their water bowls, with no thought whatsoever of the consequences (dirty water bowls). If the water bowl is dirty, then they will not drink from it, and that will in turn affect their health and growth.