The second son of the organist Vojtěch Hřímalý (1809–1880), Jan studied violin at the Prague Conservatory (1855–1861), and went on to become leader of the Amsterdam Orchestra (1862–1868). In 1869, he was appointed violin teacher at the Moscow Conservatory, where in 1874 he succeeded his father-in-law Ferdinand Laub as professor of violin studies.
It was at the conservatory that he became acquainted with Tchaikovsky. He also played at the premieres of Tchaikovsky's String Quartet No. 1 (1871), String Quartet No. 2 (1874), String Quartet No. 3 (1876) and the Piano Trio (1882), and was leader of the Russian Musical Society Orchestra in Moscow from 1874 until 1906.