After his father was arrested by the OGPU on November 6, 1937, Granovsky's mother and siblings lived in social isolation and abject poverty, shunned by their neighbors in their Moscow apartment building. Though at first unable to find work, the teenage Granovsky was eventually employed as a turner's assistant in March 1938. In May, however, he was repeatedly questioned by the NKVD about his supervisor. The NKVD apparently wondered why the supervisor would give a job to the son of a man accused of crimes against the State. As a result of the interrogations Granovsky was fired, and in October 1938 his family moved from the apartment building to a room in a ramshackle house.
On January 27, 1939, Anatoli Granovsky loitered slowly around Lenin's mausoleum in the Red Square and told a plainclothes NKVD man that he wished to be taken to the local GUGB chief. Placed in Butirki Prison under deplorable conditions, Granovsky was interrograted repeatedly, often being beaten. He would be forced sign his name on blank sheets of paper, later to find his "confession" typed above it. On the night of April 20, 1939, Granovsky was transferred to a comfortable cell in Lubianka Prison in Moscow. After indirectly denouncing his father, Granovsky was recruited into the NKVD on July 20, 1939, to spy on his acquaintances among his father's former colleagues. Following the horrifying conditions of the prisons, Granovsky was immediately given money, a salary, and exemplary treatment, including medical operations to undo the effects of illness and injury borne on him by the interrogations and imprisonment.
Anatoli Granovsky underwent numerous intensive trainings as an NKVD agent, and was assigned to the army as a cover, serving as an officer in Anti-Aircraft Defense in Moscow. During this time he worked as an NKVD counter-intelligence agent, reporting individuals for corruption or other misuse of State resources. He remained in Moscow during the German invasion of the USSR and met a young woman named Shura during the aftermath of a German bombing raid, quickly becoming infatuated with her. They began a relationship together, with Shura eventually revealing herself as a fellow NKVD agent. The relationship continued until Granovsky was informed of Shura's death.
As the Soviets went on the offensive in the second half of the war, Granovsky was trained and prepared for an operation to provide false information to the Germans: He would parachute into German territory with other Russian soldiers and then (unbeknownst to his fellows) kill them, surrendering to the Germans as a disillusioned Russian defector, allowing him to deliver misinformation.
The mission never occurred, and Granovsky instead worked with Soviet and Soviet-allied guerillas behind German lines in eastern Europe as an assassin and saboteur, causing havoc.
In spring 1945 Granovsky was assigned to begin preparing to infiltrate the West after the end of World War II, particularly the United States, by posing as a refugee with his wife, an alcoholic restaurateur whom he was to seduce and marry. The woman was eventually deemed an unreliable liability and Granovsky was reassigned to the military front.
When Berlin fell, Granovsky was one of the NKVD men who appropriated files and supplies from the Gestapo and other German agencies and sites, including the Reich Chancellery, transferring such resources back to Moscow and Kiev. Granovsky was appalled at the conditions he saw upon Germany's defeat, especially the treatment of both Germans and captured Russians and eastern Europeans at the hands of the triumphant Soviet military: Germans were beaten and raped and Soviet P.O.W.s were often tyrannized and sent to the gulags for surrendering during the course of the war.
Anatoli Granovsky was reassigned to appear as a sympathetic Soviet war veteran "fighter-poet" on tour in London, getting acquainted with the West and spreading pro-Soviet propaganda.
Later, working in Prague, Granovsky was charged with helping ensure de-Nazification and the reliability of the local Communist Party and police apparatus.
Granovsky, after working in eastern Europe, was reassigned a new cover as a member of the Merchant Navy in summer 1946. In Odessa Granovsky had been approached by the MGB, successor to the NKVD, and asked to be their spy aboard the ship Petrodvorets, with which he would rendezvous in Stockholm after traveling aboard another ship. On July 10, 1946, Granovsky arrived in Stockholm. On July 21 he slipped away from his colleagues in a crowd and went to see the assistant U.S. Military Attache. Because he could not speak English and the American could not speak Russian, they went the supervisory Military Attache, a Colonel who spoke German (in which Granovsky was experienced).
The Americans refused to grant Granovsky asylum and he was arrested by Swedish authorities. On November 8, 1946, shortly before he was to be repatriated to Soviet authorities, which was a veritable death sentence, Granovsky was granted asylum by King Gustaf V of Sweden.