"Deer are appalling. They ruin pastures, fences and the bush," Dargo walnut and wool grower Norm Wilkinson said.
Mr Wilkinson is among the farmers supporting an open letter calling on the Victorian government to reclassify deer as pests rather than as protected game.
The letter was signed by over 90 Landcare organisations, leading ecologists, agricultural groups and a range of other affected organisations and groups from across the state.
Drafted by the Victorian National Parks Association (VNPA), it asks for a significant ramp up in control measures, including serious investment in professional pest management programs.
The VNPA's Phil Ingamells says the draft deer management strategy released last year was inadequate, short on detail and relied largely on recreational hunting to solve the problem.
"There are around one million deer in Victoria and that population will be increasing at the rate of around 30 per cent each year," Mr Ingamells said.
"This is a situation that has to be addressed as a matter of urgency, with a highly strategic, professional approach.
"It cannot be controlled by ad hoc recreational hunting."
Mr Wilkinson, who has farmed in Dargo for 45 years, agrees.
"Thirty years ago, it was a novelty to spot a deer," he said.
"Recently, I've seen 20 deer in a spotlight shone from my house.
"Numbers are rising despite shooters - game shooting never controls a pest.
"There's a paddock that I haven't grazed for years and it's as short as a bowling green, thanks to kangaroos and deer.
"The bush has taken on a semi-translucent look because they've stripped any leaf, plant or bark from it as high as they can reach.
"Nothing is growing from ground level up to about six feet.
"Revegetation is impossible without two-metre-high steel mesh guards."
Deer numbers have also become unmanageable around Ensay, according to cattle farmer Peter Fraser.
"The deer population exploded after the bush fires in 2003 and 2006," Mr Fraser said.
"There's a power of deer in the bush - a shooter killed 30 in one Saturday night alone."
Mr Fraser said he and most of his neighbours had built deer fences in response.
"The fences are expensive - at least $10 a metre - and they don't stop all of the deer but fencing them out is essential," Mr Fraser said.
"It's our third poor winter in a row and there's not a lot of feed about.
"The deer are taking everything they can and a lot of them are in very poor condition."
The open letter, titled 'Controlling deer in Victoria' was sent to Minsters Jaclyn Symes, Lily D'Ambrosio and Lisa Neville.