I heard that he was ousted by the CIA after not supporting the Vietnam war
Someone might be pulling your leg there mate. He was fired as the Prime Minister of Australia because he was unable to pass a federal budget.
Unlike the US where failure to do their jobs the governmen shuts down till they can do their job, if an Australian parliament fails to pass a budget the queen's representative, known as the governer general, dissolved the parliament and new elections are held.
While there was suspicion that the Governer General was working with the CIA as he worked for a number of CIA funded fronts, the Aussie version of the CIA , called ASIO, dismissed the idea and a journalist who also investigated the claims did not see a connection. The man him himself denied it too in private letters to the Queen which have since been released to the public.
On a side note, the CIA was unhappy with the PM as he threatened to shut a Pine Gap a US intelligence gathering base in Australia. But that is just regular old politics.
It wasn't as simple as just "he got fired".
There was a bunch of dodgy shenanigans around how the GG handled the whole affair. There is a really good ABC podcast about it.
That's missing the rest of the story. It's true, Governor-General John Kerr participated in many of these groups, both CIA linked and not, but it's important to note that those front groups aren't just some talking club, they're the primary means of asset-building that the US intelligence community uses in leveraging local actors to act in America's interest. The same clubs, charity balls, conference meetings etc. were happening all over Europe, Latin America, and East Asia, with many of them becoming pro-American Heads of State, government bureaucrats, military officers, business leaders, news magnates, religious leaders, and in some cases actual far-right terrorists (especially in Italy, Greece, Turkey, Chile, Argentina, Brazil, Mexico, Indonesia, Japan, Republic of China).
If that's not enough, in 1977, the United States Deputy Secretary of State Warren Christopher made a special trip to Sydney to meet with Whitlam and told him, on behalf of US President Jimmy Carter, of his willingness to work with whatever government Australians elected, and that the US would never again interfere with Australia's democratic processes.
Lastly, Gough Whitlam himself denied the CIA was involved for most of his life because he was unaware that the then-Head of ASIO Peter Barbour had disobeyed orders to sever ties with the CIA and worked closely with the CIA in the overthrow of Allende. In an interview 8 months before Whitlam's death, the interviewer informed him that Barbour's betrayal and Whitlam lamented. Friendlyjordies had a video on the moment of truth (https://www.youtube.com/watch?v=4Dpx4o_uaGg).
It's one of those deep politics things like the Gulf of Tonkin Incident or the Assassination of Julius Caesar (not that the Dismissal is as monumental) in that journalists and analysts of the moment have little to work from except social connections and similar ideologies of the participants, but there's no definitive evidence that would be enough to satisfy a criminal trial or even a newspaper article. Even historians today aren't definitive one way or the other on the Dismissal, but its very likely that historians a few hundred years from now wont have those biases or pretensions and will just cut to the chase and ask 'who benefits from this?'