Magdalena Andersson’s centre-left coalition looks set to narrowly lose to a bloc of right-wing parties, 176 seats to 173, with 99% of the votes counted.
Moderate Party leader Ulf Kristersson is now expected to form a government.
The bloc includes the Sweden Democrats, a far-right party that has campaigned against rising gang shootings.
The final result is still to be confirmed after a recount, which is standard practice in Sweden.
Ms Andersson accepted defeat at a news conference on Wednesday. She formally offered her resignation on Thursday.
“In parliament, they have a one or two seat advantage,” she said. “It’s a thin majority, but it is a majority.”
The close-fought election campaign was dominated by gangs, immigration and integration issues, as well as soaring electricity prices.
Ms Andersson was the Nordic nation’s first female PM when she took office last year – she quit on the first day, before returning soon after.
She has been edged out by a four-party right-wing block made up of the Sweden Democrats, Moderate Party, Christian Democrats and Liberals.
It is a momentous turning point for Swedish politics – the Sweden Democrats was once treated as a pariah by political parties, but has now won around 20% of the vote.
It vowed to “make Sweden safe again” by bringing in longer prison sentences and restricting immigration.
However the party’s leader Jimmie Akesson will not become prime minister, the BBC’s Maddy Savage reports from Stockholm, because he does not have the full support of all four parties.
Instead, Ulf Kristersson, leader of the Moderates has said he will start to form a government.
“I am ready to do all I can to form a new, stable and vigorous government for the whole of Sweden and all its citizens,” Mr Kristersson said on Wednesday.
The Sweden Democrats party was born out of a neo-Nazi movement at the end of the 1980s, but has slowly grown stronger while attempting to polish its image.
In 2019 Mr Kristersson started talks with the party to form an alliance – a move which has now changed Sweden’s political landscape.
Ms Andersson told reporters on Wednesday that she understood those concerned at the party’s growing popularity. “I see your concern and I share it,” she said.
Magdalena Andersson’s Social Democrats had governed Sweden since 2014 and dominated the country’s political landscape since the 1930s.
Sunday’s vote was one of the closest elections ever in Sweden, with thousands of overseas and postal votes needed to be counted to get a clearer picture of who the winner was.
Bloc politics decide Sweden’s future
This is a huge blow to the Social Democrats who gained votes compared to the last election, and remain Sweden’s largest party.
But here it’s bloc politics that usually decides who gets into power and the right bloc has scored a razor-thin majority.
Magdalena Andersson told reporters she had to respect the Swedish people and Swedish democracy.
Now it’s up to the right to try and form a coalition.
