San Francisco( CNN) It was expected to be another volatile skirmish between right-wing activists and counterprotesters. But by Saturday afternoon, the activists were scarce in San Francisco and hundreds of locals had taken to the streets for festive demonstrations and “victory” marches.

An anti-government “free speech” group called “Patriot Prayer” had scheduled a rallying for 2 p.m. Saturday in San Francisco’s Crissy Field, a popular park near the Golden Gate Bridge. A permit was issued for the rallying and the city banned firearms and helmets, among other items, and schemed a large police presence.

Friday night, organizer Joey Gibson announced he was canceling the event, fearing violence. He blamed San Francisco Mayor Ed Lee and House Minority Leader Nancy Pelosi, whose district includes the city, for categorizing the group as white radical and elevating tensions.