Kashiwagi was born in 1922 in Sacramento, California. He grew up in Loomis, a small fruit-growing town in Placer County, California, where his Issei parents ran a fish market. He attended Loomis Elementary school, Placer High School and Dorsey High School in Los Angeles, graduating in 1940. He also attended Japanese language school, where he did his first writing and performing.
During World War II, Kashiwagi and his family were sent to the Tule Lake War Relocation Center, an internment camp for Japanese Americans. In camp, Kashiwagi spent time reading, and joined a theatre group. When the U.S. government forced detainees to fill out a Leave Clearance Application Form, commonly known as the "loyalty questionnaire", Kashiwagi refused to answer the infamous questions 27 & 28, key questions which asked internees, after a year of unjustified incarceration, if they were willing to swear unqualified allegiance to, and serve in the military for, the same government that had forced them into the camps in violation of the constitutional rights, and, if they were willing to forswear allegiance to Japan, thereby admitting an allegiance to the enemy. Unable to answer "yes-yes," to the two questions, the government took Kashiwagi's refusal to reply as a "no-no," and he was branded a No-No Boy, and he and his family were segregated by the government as "disloyals" and were ostracized by the Japanese American community. After the passage of the Renunciation Act of 1944, Kashiwagi and others at Tule Lake renounced their U.S. citizenship under government coercion.
After the end of World War II, Kashiwagi attended UCLA. He wrote his first play in 1949 for the Nisei Experimental Group, a theatre group formed in Los Angeles. His one-act play, The Plums Can Wait, was first performed in Los Angeles in 1950, and in San Francisco and Berkeley the following year. He graduated from UCLA, receiving a B.A. in Oriental Languages in 1952.
In 1959, with the help of attorney Wayne Collins, Kashiwagi had his United States citizenship restored. Kashiwagi would later dedicate his book Swimming in the American: a Memoir and Selected Writings to Collins, "who rescued me as an American and restored my faith in America".
In 1966, Kashiwagi graduated from UC Berkeley, receiving a Masters in Library Science degree.
Kashiwagi worked at the Buddhist headquarters in San Francisco for almost eight years as a translator and interpreter, English secretary and editor. He also was employed at the San Francisco Public Library as a reference librarian in literature, Japanese language materials, science and government documents, and as a branch manager. At the Western Addition Branch Library, he started what became the largest collection of Japanese language books on the West Coast. He retired after 20 years in 1987, when he was cast in Philip Kan Gotanda's play, The Wash at the Eureka Theatre.
Kashiwagi appeared in several films, including Black Rain, directed by Ridley Scott, and Hito Hata: Raise the Banner produced by Visual Communications.
He resides in San Francisco, California, with his wife and has three grown sons, including playwright Soji Kashiwagi, leader of the Japanese American theatre troupe, the Grateful Crane Ensemble. Kashiwagi is a member of Dramatists Guild and Screen Actors Guild.