In a large pot, heat 4 inches of vegetable oil to 300 degrees. Drain the sweet potato fries and pat thoroughly dry with paper towels. Working in batches, fry the sweet potatoes until softened but not browned, about 4 minutes. Transfer to a paper towel-lined platter to drain and cool.
Heat the vegetable oil to 350 degrees. Fry the sweet potatoes again until golden, about 4 minutes. Transfer to a paper towel-lined platter to drain. Combine the brown sugar, salt, cayenne pepper in a large bowl. Add fries and toss to coat. Serve immediately.