Born to Everett and Helen Ulrich in Wewoka, OK, the family moved to Livingston when Joe was two years old. Joe attended Livingston schools through high school, where he excelled in football, baseball, basketball, and track. He held the school record there in the 220-yard dash for many years.