There is a wide chasm between casual and inappropriate, and formality isn't the issue usually.
But, the OP looks a lot like bait to me, so that's all I will add.
Nope, not baiting. May be where you're at people aren't wearing PJs or sweat pants to do groceries, but here I see them all the time. I started seeing them back in college as a lot of kids just went to classes in PJs in the morning. Heck, even our biology professor almost always showed up in the early morning class wearing sweat pants and looked like she just woke up.
Yoga pants are really picking up among women in public. I remember reading about old geezers like you complaining about that. The responses have been they don't care it's extremely comfortable to wear.
Ever since I left my engineering career and pursued my own thing, I've been wearing whatever the hell I wanted when I conduct business. Sometimes, I'd go straight from the gym to a lease signing or a business deal still wearing gym shorts and tank top. Nobody cares. I haven't lost a single business deal or good tenant because I showed up in tank top.
Science fiction writers from as far back as the 20s predicted the eventual casuality of people in the future. There was a book I read written by an author from the 40s. A man's body was somehow frozen and brought back in the 21st century. There, he found people wearing whatever they wanted or nothing at all. The woman who took him in to help him adjust to the new time would go around naked because that's what she's comfortable with.
Now, don't get me wrong. I'm not advocating for being a complete slop. I groom myself. I shave every morning. I shower twice a day. I wash all my clothes and make sure I don't stink. My clothes are actually clean and straight. I'm just not wearing formal wear out in public.
If you don't believe me that people are getting more casual these days, just start paying attention. Every time you go to a restaurant, meijer, target, etc., pay attention. A lot of people just don't care anymore. They wear what's comfortable to them. Who cares what everyone else thinks?