He became successively rabbi of Vilon in 1868, Yurburg in 1870, Zhagory and then Kovno. His fame as a preacher spread, so that in 1883 the community of Vilna selected him as its maggid.
RJJ, the Rabbi Jacob Joseph School is named after him, and a playground is named after and honors the memory of a great-grandson of Rabbi Jacob Joseph who carried his name.
The Jewish community of New York wanted to be united under a common religious authority, and although the Reform and liberal factions ridiculed the idea, the mainly Russian Orthodox Ashkenazi community sent a circular offering the post throughout Eastern Europe.
Rabbi Jacob Joseph was among those offered the prestigious position. He hesitated in coming to America, aware that there were fewer religious Jews. Nevertheless, in 1888 he accepted the challenge in order to support his family, and also because he faced severe debt in Russia. The Association of American Orthodox Hebrew Congregations—comprising 18 congregations and headed by Beth Hamedrash Hagadol—was thrilled when he accepted the position.
They attempted to create one central rabbinic authority in America to maintain order in the field of Kashrus and expand Jewish education programs. Their idea ultimately failed. Although Joseph certainly possessed the credentials needed, he was confronted with many problems, primarily diverse groups of Jews, which also included anti-religious factions and Communists.
His tenure was marked by the divisiveness of New York Jewry, and the polemic of the kosher slaughterhouses of the city. Vehemently anti-religious Yiddish newspapers such as Die Wahrheit (English: "The Truth") unleashed their wrath, spreading false and malicious rumours about the chief rabbi's personal life.
Eventually, after six years, the Association stopped paying his salary. The butchers then paid him until 1895.
Although Joseph fought a losing battle in the kosher meat and poultry industry, he managed to achieve some notable accomplishments, including the hiring of qualified shochtim, introducing irremovable seals ("plumba") to identify kosher birds, and setting up Mashgichim to oversee slaughter houses. He also took an active role in establishing the Etz Chaim Yeshiva—the first yeshiva on the Lower East Side, which was founded in 1886. (It was the forerunner of the Rabbi Isaac Elchanan Theological Seminary).
Joseph published:Le'Beis Yaakov (Vilna, 1888), a collection of homilies and novellæ.
In 1897, Joseph suffered a stroke, which incapacitated him for the rest of his life. He died at age 62 and his funeral was one of the largest in New York, attended by more than 50,000 Jews. Unfortunately, it was partly marred by a public disturbance in which a number of people were injured. Employees of R. Hoe & Company, manufacturer of printing presses, threw water, paper, wood, and iron from the upper floors of the factory at 504 Grand Street. 200 policemen responded to the call, hitting and pushing the mourners. Some of Hoe’s employees, who had been harassing local Jews for some time, joined the police in the riot and beat mourners. Jewish oral tradition blamed the anti-Semitism of both the Irish factory workers and the police. Recent historical research shows that the factory workers were mostly Germans, not Irish, and that the police were following standard practice in quelling a riot. On the whole, the police kept a tight lid on inter-group violence.
After Joseph's death, a succession dispute diluted the office of Chief Rabbi and the title was effectively worthless.
Ironically, after Joseph's death many congregations began to give him the honor which they had withheld during his life. Aside from the tens of thousands that came to see him lying on his death bed, forty rabbis gathered in the cemetery for the funeral. Each one vied with his colleague to give him a better eulogy.
The congregations also competed with each other, each one desiring to bury him in its own cemetery. Congregation Adath Israel on Elridge Street promised to give his widow $1,000 on the spot and $10 a week all the rest of her life. Congregation Beis HaMidrash HaGadol was permitted to bury him in their plot at the Union Field Cemetery in Ridgewood, Queens. This became a good business venture, for the plots near the grave of the chief rabbi became extremely valuable. The widow received the amount promised for several years, and then they stopped sending her the money.
A playground on Manhattan's Lower East Side, bounded by Henry and Rutgers Streets, is named in memory of Captain Jacob Joseph (1920–1942), great-grandson of Rabbi Jacob Joseph. Captain Joseph was a member of the U.S. Marine Corps during World War II, and scion of a family devoted to religious education and civic affairs.
Born and raised in New York, Joseph left Columbia University as a junior in 1938 to enlist in the Marines. Joseph died in action at Guadalcanal on October 22, 1942. Five years later, a local law named this playground in his honor. The dedication ceremony was attended by Mayor William O’Dwyer, Parks Commissioner Robert Moses, Councilman Stanley Isaacs, and Captain Joseph’s father Lazarus Joseph – a Democratic Party leader who was a six time State Senator and New York City's Comptroller at the time.
NYC Department of Parks and Recreation also unveiled a bronze commemorative plaque on the flagstaff, which celebrates the life and bravery of Captain Joseph. This playground was built in part to meet the needs of the Rabbi Jacob Joseph School, located at the time on Henry St. The playground serves as a lasting memorial to a World War II hero, as well as to notable members of the Joseph family who have contributed to the surrounding neighborhood and to the larger New York City community.
There is a Captain Jacob Joseph Memorial Chapel at Camp Keowa, which is part of the Ten Mile River Boy Scout Camp. The Chapel was restored by a group of veterans led by Scoutmaster Tom Maher. Each August a memorial service is held for Captain Joseph at the camp. The memorial includes an interfaith service during which a Kaddish is said for Captain Joseph—who is believed to have no descendants.