The ancient Olympic Games were held in Olympia, Greece, from the 8th century BC to the 5th century AD.
Baron Pierre de Coubertin founded the International Olympic Committee (IOC) in 1894. The IOC has since become the governing body of the Olympic Movement, whose structure and actions are defined by the Olympic Charter.
Baseball has in fact chased an Olympic Dream from the beginning. Legend tells that during the 1894 Games in Athens (the first of the modern era) top USA runner Thomas Curtis gave a demonstration of the game of baseball to King George I of Greece. Curtis, trying to hit an orange with a stick, ended up spilling some juice on His Majesty’s clothes.

The first attempt to have baseball become an Olympic Sport was made 6 years later. The 1900 Summer Olympics were an international multi-sport event which was celebrated in Paris, France. No opening or closing ceremonies were held; competitions began on May 14 and ended on October 28. Baseball was not admitted (cricket and basque pelota were preferred).

Baseball made its debut at the 1904 Summer Olympics, an international multi-sport event which was celebrated in St. Louis, Missouri, in the United States from July 1 to November 23 at what is now known as 'Francis Field' on the campus of Washington University in St. Louis.
St. Louis organizers repeated the mistakes made at the 1900 Summer Olympics in Paris. Competitions were reduced to a side-show of the World's Fair and were lost in the chaos of other, more popular cultural exhibits.

Baseball was featured as a demonstration sport (A demonstration sport is a sport which is played to promote itself, most commonly during the Olympic Games; actually no demonstration sport was admitted until 1912).
In 1912 Stockholm, a United States team played against the swedish team Vastaras, winning 13-3 despite the fact an american pitched for the hosts.

In the “Inter Allied Forces Olynpics” in 1919 (June 22nd-July 6th) the USA  beat Canada 12-1
 
A huge effort was done in order to have baseball in the program of the 1936 Olympics in Berlin (Germany). Basically, the credit goes to future International Federation founder and first President Leslie Mann. Actually, both the Major Leagues (who chose Babe Ruth as their testimonial) and the japanese leagues offered their contribution to the cause. But when diplomatic relations between the USA and Germany deterioreted, the Major Leagues withdrew their support and Japan refused to compete against a team not supported by the Major Leagues. Mann selected 2 United States teams: World Champions and US Olympics. They played against each other, with Adolf Hitler himself watching the game in right field foul territory and over 90,000 spectators in attendance.

Baseball was on the program of the 1940 Olympics in Tokyo (a 9 team tournament), but the Games were cancelled by the second World War.
The 1952 Helsinki Olympics hosted, as a demonstration event, a modified form of the game: Finnish baseball, played by two local teams.
Finally, baseball was back in the Olympics when Australia played a one-game exhibition against the United States in 1956 in Melbourne (in front of a crowd of over 100.000 fans)

Thanks to the relations the Head Coach of the University of Southern California Rod Dedeaux had with with Japan (he had manager to take a team to compete there, opening the US-Japan Collegiate Series, that has been played up to these days) and the effort of the recently born National Commission of American Baseball, the old game was named a demonstration sport in the 1964 Games in Tokyo. A talented Team USA beat Japan in front of 50,000 spectators.

After a 20 year hiatus, Olympic baseball  was labeled again a demonstration sport by the IOC in the 1984 Games in Los Angeles. 
Rod Dedeaux was again the key person, since he succeeded in involving in the project Los Angeles Dodgers owner Peter O'Malley, who guaranteed with the Organizing Commitee (chaired by Peter Ueberroth) against losses. It was the first time baseball made it to the Olympics with tournament formatting.The Americas (USA, Canada, Dominican Republic, Nicaragua), Asia (Japan, Korea and Chinese Taipei) and Europe (Italy) were represented. 
Japan surprisingly won against Team USA in the final.
The tournament was such a success that the IOC voted baseball a medal sport on October 13 1986. 

In 1988 Seoul hosted another demonstration tournament. This time 5 Continents were represented, since Australia was amongst the nations invited. The United States won over Japan.

Cuba won the first 2 gold medals: dominating in 1992 in Barcelona (they won the final 11-1 against Chinese Taipei) and in 1996 in Atlanta (13-9 against Japan).
Professional players were admitted to the Games for the first time in the Sydney 2000 Olympics. A Team USA managed by Hall of Fame coach Tom La Sorda defeated Cuba to claim the gold medal. Ben Sheets threw a 4 hitter in the final, but the tournament included more stars: Cuba's Jose Contreras, Japan's Daisuke Matsuzaka, Australia's David Nilsson, Italy's Jason Simontacchi, The Netherlands' Rob Eenhoorn.

After the IOC decided to limit the program of the Summer Olympics to 28 sports starting the 2012 Games, baseball was voted out of the program on July 7 2005.
So far, the 2008 Bejing Games are the last with a baseball tournament on the program.

In August 2009 the IOC picked golf and rugby 7 over baseball for the 2016 Games.

Olympic reinstatement has been made a priority by Riccardo Fraccari after he was elected to the IBAF presidential chair in December 2009.