Photo Credit:AP
It was his first official trip outside the Kyiv region since the Russian invasion began on 24 February.
He was pictured in a bullet-proof vest surveying ruins in Kharkiv city.
Mr Zelensky told soldiers, "I want to thank each of you for your service." Later, he sacked the local security chief for "not defending" the city.
Russia has resumed shelling Kharkiv - Ukraine's second-largest city - in recent days after a fortnight of comparative quiet.
The city - Ukraine's second-largest - faced intense bombardment in the war's first months, with images showing tower blocks flattened to rubble.
Russian troops were gradually pushed back from its surrounding towns during April and May, prompting some who had fled Kharkiv to return, and the metro to reopen for the first time since the invasion.
But the city remains within range of Russian artillery. Several loud explosions were heard in the city after Mr Zelensky's visit.
President Zelensky's office posted a video on Telegram with the caption: "2,229 buildings have been destroyed in Kharkiv and the region. We will restore, rebuild and bring back life. In Kharkiv and all other towns and villages where evil came."
The footage featured Ukrainian soldiers giving Mr Zelensky a tour of the damage wrought by the war, including broken down military vehicles by the side of a road.
"In this war, the occupiers are trying to squeeze out at least some result," Mr Zelensky posted later.
"But they should have understood long ago that we will defend our land to the last man. They have no chance. We will fight and we will definitely win."
The president said he had fired Roman Dudin, the chief of the Security Service of Ukraine for the Kharkiv region "for the fact that he did not work on the defence of the city from the first days of the full-scale war, but thought only about himself".
"On which motives? The law enforcement officers will figure it out," he added.
A morale boost for Kharkiv, but risk remains
Analysis by Joe Inwood, BBC News Ukraine Correspondent
Today's visit to Kharkiv is the first time President Zelensky has left the Kyiv region since the start of Russia's invasion.
He met the troops who have not just defended Kharkiv, but repelled the Russians, almost pushing them back to the international border.
The fact that he was able to visit shows just how much the tide has turned in Ukraine's second city. A few weeks ago, this morale-boosting visit would never have been possible.
But, do not be mistaken, Kharkiv is only safe for now. Elsewhere, Ukraine very much appears on the back foot, with Russian artillery barrages forcing troops to retreat in the Donbas.
With its charismatic president, Ukraine has very much won the informational war. Unfortunately for Ukrainians, that is not where this conflict will ultimately be decided.
The president met Kharkiv's mayor and the regional governor to discuss reconstruction plans, reportedly urging them to "find cool projects" to help rehabilitate the ravaged city.
"This is a chance for such districts to have a new face," Mr Zelensky said.
His chief of staff, Andriy Yermak, said 31% of the Kharkiv region's territory was currently occupied by Russia, while 5% had been re-taken by Ukraine after being occupied earlier.
Russia is presently concentrating its offensive on the Donbas region in eastern Ukraine, which it has vowed to seize as a key war aim.
Russian forces are trying to encircle the cities of Severodonetsk and Lysychansk. On Saturday Russia's defence ministry said it had taken the nearby town of Lyman, in Donetsk - part of the overall Donbas region.
"The situation is very difficult, especially in those areas in the Donbas and Kharkiv regions," Mr Zelensky said on Saturday in his daily address to the nation.
An estimated 15,000 civilians are still in Severodonetsk, where a local official said "constant shelling" made it hard to get in or out. The water supply is increasingly precarious.
Oleksandr Stryuk, head of the city's military and civil administration, said evacuation is "very unsafe", noting "it's isolated cases when we manage to get people out".
"Now the priority is for the wounded and people who need serious medical assistance."
