If you follow the Microsoft tutorials for web development then that will incorporate EF anyway, so you can learn both together if you like. If you want to stage your learning then probably go EF first in Windows apps, then you won't have issues when you see EF used in ASP.NET tutorials.
They are actually very similar so much of your learning will apply to both. I'd say that MVC would get more use overall though, so you might want to start there. A Web API app has basically the same structure but the controller actions return JSON data that can be consumed by any application, be it web, Windows or mobile, while MVC controllers return a view for display directly in a browser. The structure of the controllers and everything below that is the same though.
That's the way .NET is going so I would suggest so. The principles are the same either way and most of the programming is the same too. There are some significant differences between ASP.NET and ASP.NET Core so definitely go with the latter, but you can still target .NET Framework with an ASP.NET Core app if you want to. I'd suggest going with the latest, i.e. ASP.NET Core 5.0 on .NET Core 5.0, but be aware that you may have to drop back to an older version of something if specific projects require it.