Sinatra was so far out of the Washington loop that he was only invited to the White House 2 times, and only when Jackie Kennedy was out of the White House, as she has a strong disliking of Sinatra. On these limited visits, Sinatra was asked to use a side door to the White House, since Kennedy didn't want the press seeing the crooner on the grounds. Frank never flew on Air Force One, was never invited to any of the Kennedy state dinners or taken to Camp David for any of the parties there but did, however, call Sinatra on an irregular basis, but this was mostly to cover the President's favourite topic, Hollywood gossip. Official wiretaps evidenced that the Kennedy brothers had resolved to severe the embryonic cord with the mobsters whom Joe -- and they -- had relied on to get them where they were. Their first step had been to cut all social ties with anyone associated with the Outfit; even Frank Sinatra, Jack’s good friend, was no longer welcome at the White House.

The Kennedys had been distancing themselves from Sinatra for some time before they gave him the axe, in part due to the singers often erratic public, and private, life that reflected badly on the family, being associated with such a person as Sinatra. The first signs of trouble came back during the election, when Sinatra hired blacklisted writer Albert Maltz to write the screenplay for a film called "The Execution of Private Slovik". The media had a field day, and because of Sinatra’s association with Maltz, accused Kennedy of being soft on communists prompting Joe to give Frank an ultimatum – drop Maltz or you’re out.

Kennedy’s advisors rightly felt that he should not be sharing a bed with Judy Campbella woman also sharing a bed with Sam Giancana, particularly when RFK was working so hard to rid the country of organized crime. The president dumped Campbell and distanced himself from Sinatra, which was long overdue in the eyes of his advisors. Dropping Sinatra wasn't a tremendous loss for the White House, they had gotten what they wanted out of Frank, and, if they ever needed him again, they knew that all they would have to do would be to snap their fingers and he'd come running due to his need for respect from the Kennedys.

Always aware of their place on the historical record, the Kennedys justified dropping Sinatra, by having one of Robert's employees at the Justice department suddenly "discover" that Sinatra had ties to organized crime, by reading a Department of Justice report about extortion in the movie business which mentioned Sinatra. To be absolutely certain that Sinatra, and everyone else, understood that he had been axed, the decided to humiliate him publicly. Towards the end of January 1962, John asked Peter to ask Sinatra if he could stay at his Palm Springs home in March while he was out west for a fund raiser. Sinatra agreed and began massive renovations to his estate, including building separate cottages for the secret service and installing communications with twenty-five extra phone lines and a huge helipad with a pole for the President's flag.

When the Kennedy’s decided that Sinatra had bragged and boasted enough to all of Hollywood that he would host the President, Jack told Peter that he couldn’t stay at Frank's place while Bobby was handling the investigation of Giancana, and gave Peter the job of telling Sinatra and finding him somewhere else to stay. Lawford was terrified of the thought of calling Sinatra with the bad news, as Lawford knew his temper. Frank was furious, lashing out at Bobby and blaming the secret service and security reasons for the change in Kennedy's plans. George Jacobs confirmed that when he got off the phone to Peter he went outside with a sledgehammer and started chopping up the concrete landing pad of his heliport. Things went from bad to worse when Sinatra learned that Kennedy was staying at the home of Republican Crooner, Bing Crosby. Sinatra, according to Lawford, "telephoned Bobby Kennedy and called him every name and a few that weren't in the book. He told him what a hypocrite, that the mafia had helped Jack get elected but weren't allowed to sit with him in the front of the bus."

Sinatra’s relationship with Giancana began to sour as a result of this, as with Sinatra out of Kennedy’s good books, it became clear that Sinatra wasn’t going to get him what he wanted (access to the White House) so the mobster had no use for him.

Sinatra was angry and wanted revenge. Sinatra thought he had exacted revenge by becoming a Nixon Republican and golfing buddy to Vice President Spiro Agnew. But then both were thrown out of office, and Sinatra ended up having crooked friends in high and low places.