With Frank’s help in eliminating Humphrey and Nixon from the 1960 presidential race by rigging the election, Frank felt as though he had proved his loyalty to JFK and the Kennedy family by asking for Giancana’s assist with crucial votes in West Virginia and Cook County, Illinois on his behalf rather than Giancana’s involvement be a favour to Jack or the Kennedy family.

Both Frank and Giancana were willing to oblige, with both of them expecting something in return. When Kennedy won, Giancana expected the new administration to show its thanks by getting the feds off his back and Frank expected respect and friendship among the Kennedy’s for his help not only in the election but for his help by performing at fundraisers during the campaign and personally planning the star-studded inaugural gala after Kennedy’s election. Both Frank and Giancana’s expectations were not met. The investigations into organized crime by RFK were still going ahead, and Frank wasn’t being invited to the White House, invited to Camp David, or invited to fly on Air Force One.

Just how far out of the Washington loop Sinatra really was, was underscored by Peter Lawford when he said that "During one of our private dinners, the President brought up Sinatra and said, "I really should do something for Frank." Jack was always so grateful to him for all the work he'd done in the campaign raising money. "Maybe," Jack used to say, "I'll ask him to the White house for dinner or lunch. There's only one problem. Jackie hates him and won't have him in the house, so I really don't know what to do." Sinatra was eventually invited for lunch, but only when Jackie Kennedy was out of the White House and even then, Sinatra was asked to use a side door to the White House, since Kennedy didn't want the press seeing the crooner on the grounds. In fact, according to Lawford, Sinatra was only allowed into the White House twice during the three years of the Kennedy administration, and then only for brief visits. "I don't think he wanted," said Lawford, "reporters to see Frank Sinatra going into the White House, that's why Frank never flew on Air Force One, and was never invited to any of the Kennedy state dinners or taken to Camp David for any of the parties there." Kennedy, or "Our Mister Prez" as Sinatra called the new Chief Executive, did call Sinatra on an irregular basis, but this was mostly to cover the President's favourite topic - Hollywood gossip. Sinatra sensed he wasn't wanted around the White House and asked why he was being pushed to the side, only to be told by the President's staff that the Kennedy brothers' wives said that they were attending too many "Sinatra summit meetings" and their wives were not happy about it. The Kennedys had been cooling off to Sinatra for some time before they gave him the axe, in part due to the singers often erratic public, and private, life. The first signs of trouble came back during the election, when Sinatra hired blacklisted writer Albert Maltz to write the screenplay for a film called "The Execution of Private Slovik". The media, the public and virtually every civic group in the country attacked Sinatra for hiring Maltz, but the ever feisty Sinatra refused to back down, in large part because he was doing the right thing, and in some part, because he was, simply, a man who wouldn't be told how to live his personal life. Boston's Cardinal Cushing, a close friend of the family, told Joe Kennedy that his son could be hurt in the conservative Catholic vote by Sinatra's hiring a communist and Governor Wesley Powell of New Hampshire had already accused Kennedy of being soft on communists. The Ambassador called Sinatra and said, "It's either us or Maltz, make up your mind, Frank." Sinatra fired Maltz, but it didn't matter. The American Legion got hold of it and went on the attack. The New York Times wrote a long piece about it and John Wayne, then the country's leading box office producer, attacked Sinatra and Kennedy for being soft on Reds. Lawford said, "The Ambassador took care of it in the end, but it was almost the end of old Frankie boy as far as the family was concerned."

Kennedy’s advisors rightly felt that he should not be sharing a bed with Judy, a woman introduced to JFK by Frank, and also a woman dating top mobster, Sam Giancana, particularly when RFK was working so hard to rid the country of organized crime. The president dumped Campbell and distanced himself from Sinatra.

Dumping Sinatra from the White House list of favoured persons was long overdue. For years, scores of Kennedy's advisors had been after the President to end his highly public relationship with Sinatra. Dropping him wasn't a tremendous loss for the White House, they had gotten what they wanted out of Frank, and, if they ever needed him again, they knew that all they would have to do would be to snap their fingers and he'd come running.

To neutralize Sinatra, and always aware of their place on the historical record, the Kennedys justified dropping Sinatra, by having one of Robert Kennedy's employees at the Justice department suddenly "discover" that Sinatra had ties to organized crime, by reading a Department of Justice report about extortion in the movie business which mentioned Sinatra. To be absolutely certain that Sinatra, and everyone else, understood that he had been axed, the Kennedy boys decided to humiliate him publicly. Towards the end of January 1962, Peter Lawford, at John Kennedy's request, asked Sinatra if Kennedy could stay at his Palm Springs home in March while he was out west for a fund raiser. Sinatra was honoured and rushed into a massive renovations program on his estate, including building separate cottages for the secret service and installing communications with twenty-five extra phone lines and a huge helipad with a pole for the President's flag.

When everything was set, and Sinatra had bragged and boasted to all of Hollywood that he would host the President, the President called Peter into the Oval office and said: "I can't stay at Frank's place while Bobby's handling the investigation of Giancana. See if you can't find me someplace else. You can handle it Peter. We'll handle the Frank situation when we get to it." Lawford was terrified of the thought of calling Sinatra with the bad news, and when he did, Lawford, who probably didn't know why the President had changed his plans, blamed the secret service and security reasons for the change in Kennedy's plans. "Frank was livid," Lawford said. "He called Bobby every name in the book and then he rang me up and reamed me out again. He was quite unreasonable, irrational really. [His valet] George Jacobs told me later that when he got off the phone he went outside with a sledgehammer and started chopping up the concrete landing pad of his heliport. He was in a frenzy." Things went from bad to worse when Sinatra learned that Kennedy was staying at the home of Republican Crooner, Bing Crosby. Sinatra, according to Lawford, "telephoned Bobby Kennedy and called him every name and a few that weren't in the book. He told RFK what a hypocrite, that the mafia had helped Jack get elected but weren't allowed to sit with him in the front of the bus." As a result, Frank ceased communication with Peter for the rest of his life, as in Frank’s eyes, Peter betrayed Frank by staying with the Crosby’s rather than Frank on the president’s visit to Palm Springs.

With Sinatra out of Kennedy’s good books, wiretaps revealed Giancana’s disappointment and his low regard for the singer. When it became clear that Sinatra wasn’t going to get him what he wanted, which was access to the White House, the mobster had no use for him, and his relationship with Giancana began to sour also.