Devise Masquerade + Multitenant with act_as_tenant

I am running a multitenant app with users and administrators scoped to their own account / domain (act_as_tenant).

My use case is the following : I have a superAdmin model who has access to a super admin dashboard where he can see the administrators and users from various accounts/domains and masquerade them. This dashboard is located on the base_domain (on my local machine -> localhost)

1) I cant use masquerade_path because I need to be able to redirect to custom domains and subdomains. Thus in an initializer I created masquerade_url method :

module DeviseMasquerade
  module Controllers
    module UrlHelpers
      def masquerade_url(resource, *args)
        scope = Devise::Mapping.find_scope!(resource)

        opts = args.shift || {}
        opts.merge!(masqueraded_resource_class: resource.class.name)

        opts.merge!(Devise.masquerade_param => resource.masquerade_key)

        send("#{scope}_masquerade_index_url", opts, *args)
      end
    end
  end
end

This is working fine.
In my application_controller : before_action :masquerade!

In my super admin dashboard I then have :
masquerade_url(user, subdomain: user.account.subdomain, domain: user.account.domain)


However whenever I click on the masquerade_url link I get redirected and in the logs I get :

Started GET "/masquerade?
masquerade=BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vZWNvbW1lcmNlLXByZW1pdW0vVXNlci8xNj9leHBpcmVzX2luPTYwBjsAVEkiDHB1cnBvc2UGOwBUSSIPbWFzcXVlcmFkZQY7AFRJIg9leHBpcmVzX2F0BjsAVEkiHTIwMjEtMDUtMDRUMjA6MDQ6MTIuNjE0WgY7AFQ%3D--771680138b1d2197a849775cbf73393b5c330876&masqueraded_resource_class=User" for 127.0.0.1 at 2021-05-04 22:03:24 +0200
Processing by Devise::MasqueradesController#show as HTML
  Parameters: {"masquerade"=>"BAh7CEkiCGdpZAY6BkVUSSIyZ2lkOi8vZWNvbW1lcmNlLXByZW1pdW0vVXNlci8xNj9leHBpcmVzX2luPTYwBjsAVEkiDHB1cnBvc2UGOwBUSSIPbWFzcXVlcmFkZQY7AFRJIg9leHBpcmVzX2F0BjsAVEkiHTIwMjEtMDUtMDRUMjA6MDQ6MTIuNjE0WgY7AFQ=--771680138b1d2197a849775cbf73393b5c330876", "masqueraded_resource_class"=>"User"}
Completed 401 Unauthorized in 1ms (ActiveRecord: 0.0ms | Allocations: 524)

So Instead of login me in I always get redirected with a 401 Unauthorized. My guess is it has something to do with cookies and cross domains maybe. Im not sure as actually when I tried hosting the super admin dashboard on the same domain as the target "masquerated" user, I got the same result....