Walton County? I live in Bethlehem, part of Barrow County, and Walton County is about a mile or two down the road from me. We are approximately in the middle between Athens and Atlanta. It is about 45 minutes to Atlanta (Midtown) for work 4 days a week. It's a nice drive down 316 to Athens any hour of the day. It is true that the road back to Atlanta is a mess after UGA football games, but it is easy enough to plan for not going in that direction at those times.
Athens is not too busy most of the time, and when we are going out to eat we generally prefer to go in that direction rather than back towards Atlanta, just to get away from the craziness of Atlanta and the suburbs. Our son loves animals, and you get to see plenty of them here.