Ok, this is a bit weird.. But after digging around in the pod specs I noticed that the FirebaseOSX podspec links to a different url than the iOS pod (FirebaseOSX.framework and an older version).
So I decided to copy the framework url from the Firebase site, and change the name and try that, and it downloaded a OSX framework for me. This is really weird, because I can't for the life of me understand that I should do like that based on what I can read on the Firebase site, so I can't really consider this the official way as it doesn't really make any sense. But it seems to work for now..
As of 2.4.0 (changelog), Firebase no longer ships a single binary for iOS and OSX, but instead has multiple frameworks (Firebase.framework for iOS, FirebaseOSX.framework for OSX). Similarly, the iOS Cocoapod is named Firebase, while the OSX Cocoapod is named FirebaseOSX. This is due to several changes in our build process, such as adding bitcode support (which only makes sense on iOS).