The Freedom Riders' negative public image was the product of many factors, but two of their most obvious problems were bad timing and a deeply rooted suspicion of radical agitation by "outsiders." Set against the backdrop of the Civil War Centennial celebration, which began in April 1961, the Freedom Rides evoked vivid memories of meddling Abolitionists and invading armies. This was especially true in the white South, where a resurgent "siege mentality" was in full force during the post-Brown era. But "outside agitators" were also unpopular in the North, where Cold War anxieties mingled with the ambiguous legacy of Reconstruction. When trying to comprehend the motivations behind the Freedom Rides, Americans of all regions and of all political leanings drew upon the one historical example that had influenced national life for nearly a century: the allegedly misguided attempt to bring about a Radical Reconstruction of the Confederate South. While some Americans appreciated the moral and political imperatives of Reconstruction, the dominant image of the tumultuous decade following the Civil War was that of a "tragic era" sullied by corruption and opportunism.
Among black Americans and white liberals the Brown decision had given rise to the idea of a long overdue Second Reconstruction, but even in the civil rights community there was some reluctance to embrace a neo-Abolitionist approach to social change. Some civil rights advocates, including Thurgood Marshall and Roy Wilkins of the NAACP, feared that Freedom Riders and other proponents of direct action would actually slow the process of change by needlessly provoking a white backlash and squandering the movement's financial and legal resources. To Wilkins, who admired the Riders' courage but questioned their sanity, the CORE project represented "a desperately brave, reckless strategy," a judgment seconded by Leslie Dunbar, the executive director of the Southern Regional Council. "When I heard about all those Northerners heading south I was sure they were going to catch hell and maybe even get themselves killed," Dunbar recalled many years later.