Well, that epithet is an honest one in how it evolved.
The early mobile homes were full of walls with plastic seams between sections and at corners. They had cheap windows, cheap linoleum, cheap carpets. The kitchens featured particle board (pressed sawdust) cabinets clad in paper veneers and with cheap hardware. The closets and hallways were small and narrow. Sometimes walls were covered in cheap wallpapers, but just as often, plastic panels.
Design accents were garish, glitzy bronze, brass, or other shiny finished plastic or aluminum trims.
They were just the housing equivalent of a Hostess Twinkie versus a Napoleon. Where building was particularly expensive, such as in the mountains or far from metropolitan areas with supplies, they made some degree of sense. They provided cheap, affordable modern housing for people who had been living in usually rural squalor, with dilapidated old houses without modern bathrooms, etc. They were built for blue-collar people.
As such, they attracted the poorest, and quickly became slum properties.
Over time, the quality of the homes improved and were used more for lake homes, or by people who had more money than the poor, but couldn't afford as large a home if they had to custom build one.
And, they began to be used in overpriced markets in big cities as affordable retirement communities. Those like Chuck lived in were actually nice places to live at one time. I saw one in Albuquerque which was indeed a beautiful neighborhood, except for the crowding, which I'm just not comfortable with. I love people, at a distance.
So, the epithet at some point migrated from condemning the relatively low quality of the materials to the neglect and abuse and condition of a mobile home once renters or recent owners had trashed it, included leaving toys out all over the property, broken mowers, piles of junk. There are most definitely people who live as trailer trash, but there are also the same behaviors in wealthy families when an old home ages out and some scion declines and can not longer pay help and begins to hang on to a property without maintaining it. See Cold Comfort Farm.