Yarkoni was born Abd el-Majid Hidr (Arabic: عبد الماجد حيدر) in the Bedouin village of Na'ura (east of Afula, in the Gilboa region of northern Israel). His family were Muslim Bedouin of the Mazarib tribe.
In 1936, at the age of 16, he joined a band which sabotaged the Trans-Arabian Pipeline (Tapline), an oil pipeline running from the Iraqi oilfields to Haifa while Palestine was under British Administration (after Israeli independence, the Tapline was redirected to Sidon, Lebanon). Due to an internal disagreement within the band, Abd el-Majid fled to a Jewish area of Palestine, where he was sheltered by Jews. There, he made contacts in the Haganah through Moshe Dayan.
Abd el-Majid kept in touch with Dayan up until the War of Independence. In December 1947, as the war raged, he placed his destiny with the Jews, joining the nascent IDF and changing his name to Amos Yarkoni.
Yarkoni served in a number of units during the War of Independence, eventually settling in the Minorities Unit, and proved himself to be an excellent tracker and patrolman, continually astounding his fellow soldiers with his abilities. In 1953, Yarkoni passed the Officer's Course (his main concern being that the exam was in Hebrew), and went on to become the Commanding Officer of the Minorities Unit.
In 1955, the frequency and effectiveness of Fedayeen attacks being launched from the Egyptian-controlled Gaza Strip and the Jordanian-controlled Hebron Hills area against isolated Israeli civilian communities increased, and the IDF searched for new ways to eliminate the threat. Rehavam Zeevi, then a senior officer in the Southern Command and in charge of securing the isolated towns of the Jordan Valley, recommended Amos Yarkoni for the task. Yarkoni and Zeevi had met some years earlier, when serving in a joint minorities/Jewish unit, and remained close friends for the rest of their lives.
Yarkoni accepted the recommendation, and the IDF tracking and counter-terrorism unit was formed under his guidance and command. The unit was named "Shaked" (Hebrew: שקד, meaning almond). Shaked was also an acronym for שומרי קו הדרום (Shomrei Kav ha Darom, Guardians of the Southern Border).
The unit fielded both Jews and non-Jews, yet when it came to Bedouin recruits, Amos was careful to accept only Bedouins from northern Israel (rather than the south of the country), so that they would not have to fight against their tribal brethren. Yarkoni was based in the Negev for many years, and was involved in countless operations. He was wounded many times, and his body was peppered with bullet and shrapnel wounds. He lost his right hand in combat in November 1959, and was badly wounded in the leg in an explosion.
In 1961, after his recovery, Yarkoni was re-appointed as Commanding Officer of the Shaked Battalion. At the time, the Military Commander commented that "if Moshe Dayan could be the Ramatkal (Chief of General Staff) without an eye, we can have a Battalion Commander with a prosthetic hand".
Even by this stage, many Israelis did not know that Amos Yarkoni was not Jewish: most simply assumed he was a Jew from an Arab country. A running joke among those who did know was "Fouad the Jew and Amos the Bedouin", referring to the two lead officers of counter-insurgency in the Jordan Valley: "Fouad" Ben-Eliezer, an Iraqi-born Jew with an Arabic name, and Amos Yarkoni, a Bedouin with a typically Sabra/Israeli name.
During the Six-Day War (1967), Yarkoni served on the Sinai front. After years of distinguished service, Yarkoni retired from the IDF in 1969.
Yarkoni died on February 7, 1991, after a prolonged battle with cancer, aged 70.
His old friend, Rehavam Ze'evi, requested that the IDF bury Yarkoni in a military cemetery, despite the fact that as a retired soldier, he was technically ineligible for a military burial. Both then-Minister of Defense, Moshe Arens, and the Head of Manpower for the IDF approved the request. In a mark of respect to Yarkoni, and going against military tradition, Yarkoni's coffin was carried by soldiers of a higher rank than him: six Brigadier Generals – his former commanding officers – carried his coffin. Behind the coffin marched the President of Israel, Chaim Herzog, political and military leaders, and ordinary Israelis from all walks of life. Also included in the cortège were former members of the Shaked Battalion.
Streets in several towns are named for him (including in Beer Sheva and Haifa).
Because Yarkoni was not Jewish, there was a problem with burying him in the Military Cemetery at Kiryat Shaul. (According to Orthodox Jewish burial law, Jews must be buried among Jews.) There was an outcry over the unseemliness of the impediment to burying a hero of the stature of Amos Yarkoni in a Military Cemetery. The matter was resolved quickly when Rehavam Zeevi, Yarkoni's lifelong friend and comrade, purchased a grave in the cemetery for himself, allowing Yarkoni to be buried at the end of the row, and therefore technically not in the Jewish part of the cemetery.
The incident over Yarkoni’s burial, and the increasing numbers of non-Jewish soldiers (especially after the influx from the former Soviet Union in the early 1990s) has seen the system of burial for non-Jews in Jewish military cemeteries streamlined.