Yes, you are correct that gravitational force must be present between two huge galaxies. We can't say that all the galaxies are moving away. For example, our neighbouring galaxy (Andromeda Galaxy) is attracted to milky way galaxies.
But many other galaxies(which are far enough that they cannot affect much by gravitation) are moving away from each other because space is itself expanding. Thus making them further apart.