Can I let only some people use the VPN, but everyone use the office network?

Yes, and that is a separate rule from the one covering the offices. On the gateway's card, tick Only allow access from Entra groups and name the groups. Only members of those groups get through that gateway; everybody else is unaffected at their desk.

The two rules are both applied, not either-or. If the client already restricts network access to certain groups, somebody connecting through the VPN has to satisfy that rule and this one. Adding a VPN gateway can therefore never widen who reaches the network, which is the opposite of what somebody restricting a VPN to five people would want.

Leaving the tick box off is the default and means this gateway adds no restriction of its own — not that nobody may connect. Membership is read from the same directory answers the network authentication uses, refreshed on a schedule, so a group change takes effect within minutes rather than at the next certificate renewal.

The restriction applies to people. A device certificate identifies a machine, which has no group membership to check, so those are reported rather than judged — use user certificates for the people connecting to a VPN.