A third of girls are not at primary school said Human Rights Watch, as it challenged the government to do more on the issue.
One in three girls and one in five boys miss out on primary school in Pakistan, according to campaigners.
They urged the new government to live up to promises to build more girls' schools despite attacks by militants opposed to female education.
Nearly 22.5 million of Pakistan's estimated 50 million children are out of school - most of them girls, Human Rights Watch (HRW) said in a report highlighting problems of poverty, lack of federal investment and a shortage of government schools.
"The number of out-of-school children in particular, and girls specifically, is on the rise and the number of government schools are not increasing in the same proportion," Saroop Ijaz, a lawyer with HRW told the Thomson Reuters Foundation.
At around the age of 14, only 13% of girls are still in education, he said. Ijaz attributed this mainly to a shortage in secondary schools for girls, as well as sexual harassment, early marriage, gender discrimination and abusive teachers.
Worldwide, more than 130 million girls are out of school, costing the global economy as much as $30 trillion, according to the World Bank.
Pakistani Taliban and allied Islamist militants, who regard girls' education as anti-Islam, have been attacking thousands of schools for young women in northwestern and northern parts of Pakistan.
In 2012, the Pakistani Taliban shot and critically wounded Nobel prize winner Malala Yousafzai, known for her girls' education advocacy in northern Swat valley.
Pakistan spent less than 2.8% of its GDP on education in 2017 - falling far short of the United Nations' recommended 4% to 6% - the rights group said.
Before his election as prime minister in August, Imran Khan promised to "prioritise establishment and upgradation of girls' schools and provide stipends to girls and women for continuing their education" in his party's manifesto.
But that has hardly happened, said Baela Raza Jamil, chief of Idara-e-Taleem-o-Aagahi, a charity that promotes access to education in Pakistan, urging the government to provide more transport, meals and uniforms to girls.
"Lack of facilities at post primary (level) and puberty leads to early marriages," she said, in a country where the United Nations says 21% of girls are married before the age of 18.
HRW interviewed some 200 people – mostly girls who never went to school or could not complete their education, and their families - and found most complained about high tuition costs, insecurity and corruption.
"The government does not help the poor. We cannot educate our children and we cannot feed ourselves," Rukhsana, a 30-year-old mother of three out-of-school children, was quoted as saying.
What some of the girls said...
A lot of times the teacher showed up late or he would not show up at all. We would just go and sit and then come home.
My parents said, ‘If you are interested enough you can walk there.’ The path is lonely and isolated and there have been kidnappings… But then I realised I needed to study so I convinced my parents and I got friends to go so we walked to school together.”