This thesis presents some new algorithmic techniques for high-order context modeling, examines and demonstrates the efficacy of these techniques in multimedia data compression. The proposed context modeling algorithms are guided by universal source coding principle, and assume no knowledge about the source being coded. The context model is constructed by an on-line learning process which estimates the conditional probability of future samples based on the statistics of the samples being coded (the so-far observable).