Yes, in "Grail", Lennier says that there are two Minbari castes, the religious and the warrior, which implies that he doesn't consider the workers relevant in a description of Minbari society or culture... Understandable; they don't seem to play any role in shaping Minbari policy - we've never heard about what their position on the Earth-Minbari war was, for example...

I always thought it a shame we never got a major Worker-caste character, get an idea of just how happy they are to be in what is clearly a subservient role compared to the other two castes, which seem to be on more of an equal footing. I always thought it was significant that we never hear of someone switching castes either to or from the Worker-caste.