They don't have opponents directly affecting them, or jockeying for position, etc. That happens in the longer track events but not in the 100, 200, long jump, discus, etc.
Golf at least ties several elements together - the drive, putting, hitting from rough/sand/fairway. Plus it’s 18 holes for several hours.
For swimming, track and field, they’re doing the event once, for the most part, under same conditions, relatively speaking, for a few minutes max.
Same as golf, I wouldn’t consider swimming, track field or gymnastics in the same category as football, volleyball, tennis, baseball, hockey, water polo, cricket, hurling, badminton, reluctantly even pickleball. Dodgeball is in the group.
Maybe I draw the line before ping pong. Same with billiards, curling, cornhole, shuffleboard and gameday chatter. Gotta be a higher level of movement.