Cold water has been poured on calls to "conscript" young people, living with their parents and currently on JobSeeker, to work on farms.
Fruit Growers Victoria services manager Michael Crisera said growers were very concerned about the sharp drop in both backpacker numbers and visitors on working holiday visas.
"There's 1000 leaving each week at the moment, and none coming in to replace them," Mr Crisera said.
"We are looking to see if the government will revive some old projects, such as the Seasonal Workers Incentive Trial, where Australian job seekers can keep their payments and earn up to $5000 on the harvest trail."
Mr Crisera also said he'd like to see the Northern Territory trial, allowing workers from Vanuatu to come in to pick mangoes, extended to the fruit growing sector.
"When people are forced to go on a farm to work for the dole, it's not very pleasant," he said.
"It's very intermittent - you could work for 10 days and not for another 10, and you would be going to multiple farms."
Many people who had been recently laid off were from the hospitality sector or held other indoor positions.
"We need people that can work in the heat, they need to be physically fit, can climb a ladder and handle a fruit picking bag."
He said it was also unfair to try and conscript the unemployed.
"I think they are unfairly tarnished, for a few long-term unemployed people."
Shepparton orchardist Peter Hall also rejected the idea.
Mr Hall said around 140,000-150,000 backpackers visited each year, and while it was cause for concern that number had halved, they could not easily be replaced by the unemployed.
"It's been talked about for decades," Mr Hall said.
"There's always been a level of unemployment in Australia, and the real question is whether someone who hasn't done hard physical work before is fit for what is required.
"People who have been unemployed for a long time need the mental and social skills to be work ready.
"There are reasons why a person is long-term unemployed - there are social, logistical and domestic reasons why a person hasn't been able to find work."
Mr Hall said while the work wasn't overly hard, physically, it was demanding in other ways, with early start times.
"It's very simplistic to say they can all pick fruit," he said.
The nature of fruit picking had also changed, in the last 30 years.
"The quality of the fruit we were picking wasn't particularly high; it was for processing, it didn't matter.
"These days, most of it is picked as fresh fruit, so there is a much higher standard of care required.
"It's much more of a highly skilled task."
Mr Hall said fruit growers were also looking for labor from a variety of sources, including backpackers, migrants and those under the Australian Government Seasonal Worker Programme (SWP).
With the main picking season starting in October, Mr Hall said making up the loss of backpackers would be challenging.
"There are two aspects, the planes are not coming in with backpackers, that's one, the second one is the border restrictions," he said.
"There may be some workers in NSW who want to come, but they are potentially worried, if they come to Victoria, they can't get out.
"The Victorian government has been more draconian in than other states, in terms of lockdowns, and that could scare people off."
Thomas Elder Markets analyst Andrew Whitelaw said farm work was now far more specialised, than it ever was.
'It's a big change from the real old days, when there was a lot of manual labor, and you could just get convicts on your farm, to pick some spuds," Mr Whitelaw said.
"It devalues agriculture, to get unskilled people to come on farm, and do some work.
"If people apply for those jobs, that's a different matter, that shows they have made an effort to apply, but conscripting people is not the right option."
Mr Whitelaw said farmers could not just expect someone to come on-farm and start working straight away.
"If you have a staff member, who doesn't want to be there, does it lead to safety concerns, and more instances of farm accidents, because you have unengaged workers?" he said.
The bigger issue was border closures, which needed to be resolved quickly.
"We really need some sort of leadership on this issue, I would say in the next fortnight," he said.
"There needs to be a common sense approach to the whole situation, so you can get people where they need to go."
In the grain industry, there were many northern hemisphere farmers who came to Australia to help with the harvest.
He said the other, often overlooked group, who provided casual labor were "Grey Nomads" - retired people, who were travelling around the country.
"They'll go around the country and may do a little bit of harvest work, such as working on a sampling stand for GrainCorp, or CBH, and they do that as a bit of a way of subsidising their holidays."
Border restrictions, particularly between NSW and Queensland, could also delay the free flow of grain from centres like Moree.
"If there are restrictions, it's an extra layer of complexity, and complexity costs money," Mr Whitlelaw said.
"We really need some sort of leadership on this issue, I would say within the next fortnight.
"There needs to be a commonsense approach to the whole situation, so you get movement of people to go where they need to go."
Mr Whitelaw said a 2018 Australian Bureau of Agricultural and Resource Economics study found the productivity of seasonal workers was higher than that of working holiday makers, when it came to fruit picking.
"This estimate was derived from data for 150 seasonal workers and 109 working holiday makers over three years," the ABARES report found.
"Seasonal workers who returned to the farm were on average 15 per cent more productive than new seasonal workers.
"This is mainly because returned workers required minimal induction and training in subsequent seasons as they had previously acquired skills and farm knowledge."
A key consideration for many growers was the reliability of labour during the critical period of harvest.
" A priority for growers was for the harvesting process to run smoothly with minimal disruption," ABARES found.
Federal Agriculture Minister David Littleproud said the commonwealth was working with state and territory governments to address the workforce challenges presented by coronavirus.
"We have been alive to these issues since the start of the pandemic, extending visas for overseas workers involved in agriculture and facilitating the movement of these workers to states where they are needed," Mr Littleproud said.
"The Northern Territory Trial reopening of the Seasonal Worker Program and Pacific Labour Scheme highlights were willing to be agile to meet the needs of the farming sector.
"States have a critical role to play in all this and I want to see a more commonsense approach to interstate borders to make sure that the workers that are able and willing to take up these jobs can do so."
Have you signed up to Stock & Land's daily newsletter? Register below to make sure you are up to date with everything that's important to Victorian agriculture.
Sign up for our newsletter to stay up to date.