We think of the sensor as something divorced from the rest of the system (and ignore light actually has a wavelength). Given a specific aperture, diffraction will impact a smaller sensor more than a larger one. Larger sensors don't need to work at such high frequencies and so can take better advantage of optics optimized for contrast. And given the same angle of view with a given aperture, the larger format will also have greater resolution (but less DoF) than a smaller sensor because of a larger entrance pupil.
We like to think of these arguments on the terms of everything being equal. The problem is everything is different. We can try to push and shove some of these criteria around to try to make them equal, but we are just creating greater difference with other factors. Ultimately, it does not matter about the individual technical specifications, but the symphony for factors that goes into the image we perceive.