Metallic lead is not a problem, only lead vapor or salts. The vapor comes from casting and is inhaled, the salts from primer residue and is mostly ingested, with lesser amounts inhaled.
Wash your hands well after casting or any reloading procedure. This is the most important thing you can do. Refraining from eating, drinking and smoking while casting/reloading should be obvious, but I'll throw it in here anyway.
Wash all your brass before processing. This will remove the vast majority of the dirt, primer residue, etc. A 5-gal bucket, hot water and some dish soap will take care of that.
I deprime all cases before tumbling, using a Lee universal decapping die. Wear a mask (cheap at most drugstores) to prevent inhaling the dust. Used primers fall into an empty coffee can, which is then closed and discarded when full. The benchtop is wiped down with a wet paper towel afterwards.
Aside: If you live where icy roads are an issue, catch the used primers in a screw-lidded container and put it in your car trunk. They make great (free) traction compound on ice.
Throw a used dryer sheet or 2 in the tumbler along with the brass to hold onto the dirt and primer residue. That one thing markedly cuts down on the dust. Put the mask on when you dump and sift the media. Discard the media more frequently than you're used to doing. Bedding material for pet lizards is the same crushed walnut shells as tumbling media, usually much cheaper.
Assuming your lead pot is something like a bottom-pour Lee, fabricate a cover for the pot to reduce lead vapor (it also helps maintain melt temp). Ideally it would be something like the old asbestos tiles we used to use in chemistry labs. If you can't find one of those, a metal cover will work. Casting is typically too damn hot work to even think about a mask, and it mostly wouldn't help against vapor anyway. If you do any dip (ladle) casting, save that for when you can work outdoors.