A graduate of St. Laurence High School in Burbank, Illinois, just outside Chicago, Dwyer was selected by the St. Louis Cardinals in the 1971 draft out of Southern Illinois University, and he wasted little time in the minor leagues, debuting in the majors in 1973 with the Cardinals. He became known as a fastball hitter who was used mostly against right-handed pitching and played all three outfield positions well.
Midway through the 1975 season, he was traded to the Montreal Expos (1975–76). The following season, another midseason trade landed him with the New York Mets (1976). He rejoined the Cardinals for the 1977 season and part of 1978, and later played with the San Francisco Giants (1978) and Boston Red Sox (1979–80). Finally, he found a home with the Baltimore Orioles in 1981.
With Baltimore, Dwyer became a role player as a designated hitter and pinch-hitter. He enjoyed a good season in 1982, hitting .304 (74-for-260) in 71 games, but his most productive year came in 1983, when he appeared in 100 games while hitting .286 with eight home runs and 38 runs batted in, helping his team to reach the World Series, won by Baltimore in five games. In Game One of the Series, Dwyer got the Orioles only run with a solo home run off Philadelphia Phillies pitcher John Denny.
During the 1987 season he hit a career-high 15 home runs in 241 at-bats. From 1988-90 Dwyer was technically on five teams, playing with Baltimore (1988[start]) and Montreal (1989 [end]), but he did have three separate stints with Minnesota (1988[end]-1989 [start]; 1990).
For his career, Dwyer was a .260 hitter (719-for-2761) with 77 home runs and 349 RBI in 1328 games, including 409 runs, 115 doubles, 17 triples, 26 stolen bases, and a .353 on-base percentage. In four postseason games he hit .333 (4-for-12), including one home run, two doubles, four runs, and one RBI . He also collected a career 103 pinch-hits. He also played from 1977 to 1980 with the Mayagüez Indians of the Puerto Rican Professional Baseball League, and following his majors career, Dwyer played for the 1990 Sun City Rays of the Senior Professional Baseball Association.
Dwyer, who was one of the most extreme platoon players of the 1980s, started just 11 games in July 1983, playing a complete game just four times that month. He hit .452 in just 48 at-bats, ranking fifth for the Best Month Performance in the all-time list behind Larry Walker (.528 in 68 AB, September 1998 and .513 in 49 AB, September 1999), Jim Rice (.525 in 66 AB, September 1985) and Sean Berry (.524 in 49 AB, August 1998). In the heat of the 1982 pennant race, Jim reached base 13 consecutive times over 4 games against Detroit and Milwaukee (Sept 29 thru Oct 2).
Following his playing career, He coached (1991) and managed (1992–94) the triple A Portland Beavers. In 1995, he became hitting coach of the Minnesota Twins' Double-A affiliate, the New Britain Rock Cats, and has remained within the Twins' organization in one capacity or another ever since. He remained with the Rock Cats through 1996 before becoming Twins' minor league roving hitting coordinator (1997-2005). Currently, he is the hitting coach of Minnesota's advanced A affiliate, the Fort Myers Miracle (2006-present).