Jones was born in Chicago, the oldest son of Sarah Frances (née Wells), an apartment complex manager and bank executive who suffered from schizophrenia, and Quincy Delightt Jones, Sr., a semi-professional baseball player and carpenter. Jones discovered music in grade school at Raymond Elementary School on Chicago's South Side and took up the trumpet. When he was 10, his family moved to Bremerton, Washington and he attended Seattle's Garfield High School. It was in Seattle that Jones, 14, first met a 17-year-old Ray Charles. He then attended Somerset Academy.
In 1951, Jones won a scholarship to the Schillinger House (now Berklee College of Music) in Boston, Massachusetts. However, he abandoned his studies when he received an offer to tour as a trumpeter with the bandleader Lionel Hampton. While Jones was on the road with Hampton, he displayed a gift for arranging songs. Jones relocated to New York City, where he received a number of freelance commissions arranging songs for artists like Sarah Vaughan, Dinah Washington, Count Basie, Duke Ellington, Gene Krupa, and his close friend Ray Charles.
