They came, the faithful few, via places such as Dubai, Doha, Istanbul and Minsk, to the very edge of Europe, on a pilgrimage that stretched wallets, passports and, for some, even consciences. Almost 2,500 miles from London, Arsenal and Chelsea's European bandwagons had reached their final destination in Baku, the Azerbaijani capital, but only the most committed of passengers remained on board.

There is no doubt that Uefa's decision to hold the final here has tested the loyalties of Arsenal and Chelsea's supporters. Jordan Gould, an Arsenal fan, had to catch a plane from London to Minsk, then a connecting flight to Tbilisi, then get a night bus to Baku. He was one of the lucky ones.

"Some Chelsea fans tried to hire a car in Georgia to drive all the way to Azerbaijan," he said. "Once they got to the border you find out your insurance is invalid, so their car got confiscated and they were stuck at the border. They were asking if we had seats on the bus, which conveniently we didn't."

The seats on the bus may have been taken, but plenty of empty ones revealed themselves in the stands of the Baku Olympic stadium as the smoke from the pyrotechnics of the bombastic opening ceremony cleared. Best estimates suggested that about 4,000 Arsenal fans had made the trip from England, and about 2,000 Chelsea fans, with those numbers swelled by those from Russia, Turkey and Asia. Still, it was a decidedly sparse look for one of Uefa's showpiece finals.

The atmosphere in the stadium was muted, and especially at times during a low-key first half, it had the feel of a game in a pre-season tournament. The traditional fan songs, which had been a rousing chorus in Valencia and Frankfurt, were more of a murmur here, for all the best efforts of those present.

"We'd have had 25,000 if it had gone to an Amsterdam, a Paris, a Madrid, and yet here, we've sold 4,200 tickets," said Mick Gould (no relation), one of the lucky Arsenal fans who flew directly, albeit for £1,150. The contingent in the red corner of the stadium looked larger and more organised than that in the blue corner, reflecting the difference in numbers. "We walked past a place that was playing Blue is the Colour on repeat, and there were two people in there," Jordan Gould said.

Most people I spoke to were pleasantly surprised by Baku, a strange melting pot of Caucasian history, Soviet architecture and nouvelle richesse. "It wasn't what I was expecting, it's quite clean, modern, very architectural, the food is incredible," Charlie Harris, an Arsenal fan, said.

But the implications of the choice of venue were troubling to the supporters. Many were conflicted by the issue of Henrikh Mkhitaryan, the Armenian Arsenal player who did not travel to the final because of concerns over his safety arising from the territorial dispute between Armenia and Azerbaijan. "It's horrific. It shouldn't have been allowed to happen," Harris said. "You shouldn't be allowed to hold a final that could jeopardise one of the teams."

Theo Skarnr, a teenager who travelled from Sweden, said that he had had the back of his Arsenal shirt checked by an overzealous local policeman on the lookout for Mkhitaryan jerseys.

Most of the criticism was directed at Uefa, but among the flag-bearing footsoldiers in Baku, there was a cynicism that encompassed the entire football landscape. "The clubs are complicit in this: we have Visit Rwanda on our sleeves, and as much as [Arsenal] have made statements condemning the fact it's in Baku, had a group saying Visit Baku paid more, they'd happily have that on their sleeves," Jordan Gould said. "The money that's in the game, we want to see £100 million spent every transfer window – the fact that finals are held here makes that possible."

That's the strange thing about a game like this: it brings out the deepest disaffection with football, but it also evinces its unconditional attraction. I encountered four neutrals from Sheffield, here just "because we're mental" one said. "We were hoping for something a little more exotic," John Ripley said. "Benfica were in at that time."

Daniel Cornwell, another member of the group said that he felt morally torn about coming to an authoritarian state.

"What I heard on the news is a bit of a whitewash through sport, which is worrying," he said. "If that had been clearer, I would have thought twice."

They will remember their odyssey, these 6,000-odd English fans, and for all the pounds, air miles and moral compromises, next time Uefa takes them to the edge of a continent, the edge of their patience, they'll probably do it all over again. "I'm feeling smug," Jordan Gould said. "I've already got my ticket to Istanbul for the Super Cup."