The VAT location for the conference is Sweden, so this is in itself an "inner-Swedish" type of invoice (like buying a cinema ticket when you are in Sweden). (As opposed to, say, buying Swedish goods to be delivered outside Sweden - then the invoice would be of "export" type and would usually become your duty to pay your country's VAT on importing goods).
For a private person (and probably also small businesses that have VAT exemption for their sales), this is where the story ends: you are subject to Swedish VAT in that matter, and your nationality doesn't change anything about this.
However, if you were a business (including freelancer) within EU and do have a EU VAT-ID (in addition to your national VAT-ID), the EU VAT/tax treaties would allow you to file for a Swedish VAT return even though you do not have a Swedish VAT number.
This is basically a double-taxatation agreement: it avoids different taxes arising just because the conference is in a neighbour country.
Such double taxation agreements/tax treaties exist not only within the EU, but with some other countries as well
- The process probably won't be as streamlined as it is within EU (where I can use a a web site in my language provided by my national tax office to request reimbursement from other EU countries)
- But the main advantage within EU here is that as EU citizens we can just rely on this working mostly the same for all EU countries, so once I found out how this is done with a conference in Spain, I don't have to dig again as long if the conference is next in Sweden.
For non-EU businesses, you'll probably have to find out for each other country separately whether there is a tax treaty and how to apply for refund.
- Of course, you may be unlucky: there are countries with which no such tax treaty exists, and then you may not be able to get VAT refunds.
- In any case, it is the Swedish tax office that you have to ask for reimbursement. They provide some information here and it may be worth while calling them for more information.
- Academic conference organisers typically know that they have to charge VAT and that this is a particular type of service where the VAT doesn't go by reverse charge like for goods and "normal" services (and tell you the VAT law paragraph that says so), but they may not know if/how/where to apply for refund: the vast majority of their customers are from universities and governmental research institutions working on project money. Even if these entities do have VAT-IDs, the grants are usually not subject to VAT (i.e. the research institute doesn't write an invoice including VAT to the granting agency/government) and in consequence also are not eligible for VAT refunds.