Eliminating things that are the possible trouble maker may take some time and effort.
Fill valve(s)....rare to have them make dirty water, possible but rare.
Water line to the refrigerator....fairly common them make dirty water depending on water conditions in your area. Here we use a lot of copper lines which seems to give us less issues than a plastic one does. many of the newer SS lines have a plastic line inside the SS cover.
Water tank/reservoir....very high for them make dirty water.
LINK> Water tank assembly for water dispenser
LINK> How to change the reservoir
LINK> Copper ice maker water line kit
I'd kinda be inclined to try a new water line to the refrigerator and a new water tank/resevour as a good plan of attack.