No, in itself it is not rude. You can do it in a rude way, by just ignoring the signals if the other person is not happy about it.
Offer the person you talk with the choice.
I would start with a Norwegian hello or good morning/day, and next try out which language sits best.
One time in Norway I was asked to speak Dutch rather than English (I speak no Scandinavian languages) while the seller at the market kept to her version of Norwegian, it worked for us.
I have always found all Nordic people helpful and glad to communicate in whatever language or mix of languages worked best, often not their own.
While most adults in Sweden and Norway will have learned English, it is not 100% that have. And the percentage that are used to speaking English will be lower outside of the areas where tourists from abroad are common.
Whether trying to speak Norwegian or keeping on Swedish or switching to speak English is best depends on your language skills, and likely how long you have been around people speaking Norwegian and how many related languages you really know.
I have found Norwegian always the easiest of the Scandinavian languages but that might be that I learned to understand Frisian (of the Dutch province Friesland) even though I do not speak it.
Not Scandinavia but I have heard and/or been part of a lot of German-Dutch conversations. Some people, mostly the young ones, would ask for English while the rest were happy to understand each other.