Gandhi's views were not fixed but changed as he himself evolved. He was definitely not casteist in the traditional sense i.e. thinking some castes were superior to others by birth. But some of his statements were complex. If taken out of context his statements can me misconstrued as saying he supported casteism or was against blacks.
While I am not an admirer of Gandhi, it would not be correct to trivialise him as being a pedophile or castest or racist.
BTW, Gandhi was weird by Indian standards also - not just by Western standards. Most Indians who have heard about his strange experiments with naked girls are much more outraged than the more liberal Westerners.