draft-egge-videocodec-tdlt.xml 27 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699
  1. <?xml version="1.0" encoding="utf-8"?>
  2. <!DOCTYPE rfc SYSTEM 'rfc2629.dtd'>
  3. <?rfc toc="yes" symrefs="yes" ?>
  4. <?rfc compact="yes"?>
  5. <rfc ipr="noDerivativesTrust200902" category="info" docName="draft-egge-videocodec-tdlt-00">
  6. <front>
  7. <title abbrev="TDLT">Time Domain Lapped Transforms for Video Coding</title>
  8. <author initials="N.E." surname="Egge" fullname="Nathan E. Egge">
  9. <organization>Mozilla Corporation</organization>
  10. <address>
  11. <postal>
  12. <street>650 Castro Street</street>
  13. <city>Mountain View</city>
  14. <region>CA</region>
  15. <code>94041</code>
  16. <country>USA</country>
  17. </postal>
  18. <email>negge@dgql.org</email>
  19. </address>
  20. </author>
  21. <author initials="T.B." surname="Terriberry" fullname="Timothy B. Terriberry">
  22. <organization>Mozilla Corporation</organization>
  23. <address>
  24. <postal>
  25. <street>650 Castro Street</street>
  26. <city>Mountain View</city>
  27. <region>CA</region>
  28. <code>94041</code>
  29. <country>USA</country>
  30. </postal>
  31. <phone>+1 650 903-0800</phone>
  32. <email>tterribe@xiph.org</email>
  33. </address>
  34. </author>
  35. <date day="15" month="October" year="2012"/>
  36. <area>RAI</area>
  37. <abstract>
  38. <t>
  39. This proposes the use of Time Domain Lapped Transforms (TDLT) as the transform
  40. step for video coding.
  41. </t>
  42. </abstract>
  43. </front>
  44. <middle>
  45. <section anchor="intro" title="Introduction">
  46. <t>
  47. This draft outlines a proposal to adapt the Time-Domain Lapped Transforms
  48. (TDLT) for use in video coding.
  49. Lapped transforms were proposed for video coding at least as as far back as
  50. 1989&nbsp;<xref target="Malv89"/>.
  51. Like the loop filters more commonly found in recent video coding standards,
  52. TDLTs use a post-processing filter that runs between block edges to reduce or
  53. eliminate blocking artifacts.
  54. Unlike a loop filter, the TDLT filter is invertible, allowing the encoder to
  55. run the inverse filter on the input video.
  56. This decorrelates blocks before they are passed through a normal block
  57. transform and quantization step, improving coding gain (which helps in both
  58. smooth and highly textured areas), in addition to reducing blocking artifacts.
  59. </t>
  60. </section>
  61. <!--section anchor="terminology" title="Terminology">
  62. <t>
  63. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
  64. "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
  65. interpreted as described in <xref target="RFC2119"/>.
  66. </t>
  67. </section-->
  68. <section anchor="tdlt_def" title="TDLT Defined">
  69. <t>
  70. The Time-Domain Lapped Transform can be viewed as a set of pre and post filters
  71. to an existing block-based DCT transform.
  72. The idea is to place an invertible filter along the block boundaries outside
  73. an existing block-based DCT encoder.
  74. </t>
  75. <!--figure anchor="filter_diagram" title="Filter Diagram" align="center"-->
  76. <figure align="center">
  77. <artwork align="center"><![CDATA[
  78. +----+ +----+
  79. | | Q | |
  80. | P |+----+ u +----+|P^-1|
  81. | || | a | || |
  82. +----+|DCT | n |iDCT|+----+
  83. | || | t | || |
  84. | P |+----+ i +----+|P^-1|
  85. | || | z | || |
  86. +----+|DCT | a |iDCT|+----+
  87. | || | t | || |
  88. | P |+----+ i +----+|P^-1|
  89. | | o | |
  90. +----+ n +----+
  91. ]]></artwork>
  92. </figure>
  93. <t>
  94. The pre-filter P operates in the time domain, processing block boundaries and
  95. removing inter-block correlation.
  96. The blocks are then transformed by the DCT into the frequency domain, where
  97. the resulting coefficients are quantized and encoded.
  98. When decoding, the inverse operator P^-1 is applied as a post-filter to the
  99. output of the inverse DCT.
  100. This has two benefits:
  101. <list style="numbers">
  102. <t>
  103. Quantization errors are spread over adjacent blocks via the post-filter
  104. P^-1, reducing blocking artifacts.
  105. This eliminates the need for a separate deblocking filter.
  106. </t>
  107. <t>
  108. The increased support region of the transform allows it to take advantage of
  109. inter-block correlation to achieve a higher coding gain than a non-overlapped
  110. DCT.
  111. This allows it to more effectively code both smooth and textured regions.
  112. </t>
  113. </list>
  114. </t>
  115. <t>
  116. The pre-filter P is defined in <xref target="Tran01"/> as follows:
  117. <!--figure anchor="prefilter" title="Pre-Filter Operation" align="center"-->
  118. <figure align="center">
  119. <artwork align="center"><![CDATA[
  120. 1 [ I J ][ I 0 ][ I J ]
  121. P = --- [ ][ ][ ]
  122. 2 [ J -I ][ 0 V ][ J -I ]
  123. ]]></artwork>
  124. </figure>
  125. Here I is the identity matrix and J is the "reversal matrix", obtained by
  126. simply re-ordering the rows of the identity matrix in reverse order.
  127. The V matrix is a free parameter, and as long as V is invertible, this filter
  128. structure guarantees perfect reconstruction, linear phase, and
  129. biorthogonality.
  130. If V is orthogonal, then the overall transform is also orthogonal instead of
  131. just biorthogonal.
  132. </t>
  133. <t>
  134. For the case of the 4x8 TDLT, we use the following invertible matrix for V:
  135. <!--figure anchor="rotation" title="Type-IV approximation for 4x8 TDLT" align="center"-->
  136. <figure align="center">
  137. <artwork align="center"><![CDATA[
  138. [ 1 q_0 ][ 1 0 ][ s_0 0 ]
  139. V = [ ][ ][ ]
  140. [ 0 1 ][ p_0 1 ][ 0 s_1 ]
  141. ]]></artwork>
  142. </figure>
  143. Thus for the 4x8 case, the pre-filter and post-filter are completely described
  144. by the four parameters q_0, p_0, s_0, and s_1.
  145. In general, any invertible V matrix may be used.
  146. However, factoring V into a series of lifting steps ensures that it can be
  147. implemented efficiently, and can reduce the number of parameters required by
  148. the optimization process, since the full flexibility of an arbitrary
  149. invertible matrix is not required to achieve good coding gain.
  150. <xref target="Tran01"/> proposes two reduced-parameter factorizations, dubbed
  151. Type&nbsp;III and Type&nbsp;IV.
  152. These are identical in the 4x8 case, but for larger transforms the differ in
  153. the order that the p_i and q_i steps are applied: interleaved for
  154. Type&nbsp;III and ascending and then descending order for Type&nbsp;IV.
  155. While Type&nbsp;III appears to give slightly higher coding gain when
  156. unconstrained, when coupled with the ramp constraint discussed below and the
  157. constraint that all coefficients be dyadic rationals, the number of feasible
  158. solutions is much smaller than with Type&nbsp;IV.
  159. The increased number of feasible solutions allows Type&nbsp;IV transforms to
  160. achieve higher coding gains than Type&nbsp;III when these constraints are
  161. imposed.
  162. This definition easily extends to the 8x16 and 16x32 TDLT case with similar
  163. parameterizations.
  164. In general, we use the Type&nbsp;IV factorization
  165. from&nbsp;<xref target="Tran01"/>.
  166. For a V matrix of size M, this has (M-1) p_i and (M-1) q_i parameters, and
  167. M s_i parameters.
  168. For a transform of size Nx2N, this gives a total of 1.5N-2 parameters.
  169. This is also the number of lifting steps that must be performed to implement
  170. the V portion of the pre- and post-filters.
  171. </t>
  172. </section>
  173. <section anchor="metrics" title="Lapped-Transform Selection">
  174. <t>
  175. We would like to find good candidate transform coefficients that perform well
  176. within a video coding framework. There are several metrics we can use for
  177. evaluating pre-filter parameters. Including
  178. </t>
  179. <t>
  180. <list style="numbers">
  181. <t>Coding Gain - how well energy is compacted into only a few coefficients</t>
  182. <t>Side Band Attenuation - how much energy from frequencies outside the passband leaks into each basis function</t>
  183. <t>Transform Width - how wide are the basis functions and how much ringing they will cause</t>
  184. <t>Orthogonality - how linearly independent the basis functions are</t>
  185. </list>
  186. </t>
  187. <t>
  188. Of these, the most important by far is coding gain as it allows us to directly
  189. measure the improvement in bits between different candidate transforms. At
  190. high bit rates using an efficient quantizer, every 6.02&nbsp;dB improvement in
  191. coding gain saves a bit of entropy per coefficient.
  192. </t>
  193. <section anchor="coding_gain" title="Coding Gain">
  194. <t>
  195. Coding gain is a useful metric for comparing different candidate transforms.
  196. Roughly speaking, it is the measure of how well energy is compacted into only
  197. a few coefficients.
  198. The formula for coding gain of the lapped transform can be found in
  199. <xref target="Terr12"/>.
  200. Using an AR(1) model with r=0.95, we have
  201. <figure align="center">
  202. <artwork align="center"><![CDATA[
  203. / 1 \
  204. C_g = 10*Log_10 | ----------------------------------------- |
  205. \ Prod_i((G*AR(1)*G^T)[i,i]*(H^T*H)[i,i]) /
  206. ]]></artwork>
  207. </figure>
  208. </t>
  209. <t>
  210. where G is the analysis filter of the lapped transform:
  211. <figure align="center">
  212. <artwork align="center"><![CDATA[
  213. [ ][ P 0 ]
  214. G = [ 0 DCT 0 ][ ]
  215. [ ][ 0 P ]
  216. ]]></artwork>
  217. </figure>
  218. and H is the synthesis filter of the lapped transform:
  219. <figure align="center">
  220. <artwork align="center"><![CDATA[
  221. [ P^-1 0 ][ 0 ]
  222. H = [ ][ iDCT ]
  223. [ 0 P^-1 ][ 0 ]
  224. ]]></artwork>
  225. </figure>
  226. </t>
  227. <t>
  228. In <xref target="Terr12"/> the coding gain of the non-lapped DCT is compared
  229. with the optimal non-lapped Karhunen–Loève transform for the same AR(1) model
  230. with r=0.95.
  231. <figure align="center">
  232. <artwork align="center">
  233. 4 point 8 point 16 point
  234. +-----------+-----------+-----------+
  235. DCT | 7.5701 dB | 8.8259 dB | 9.4555 dB |
  236. KLT | 7.5825 dB | 8.8462 dB | 9.4781 dB |
  237. +-----------+-----------+-----------+
  238. </artwork>
  239. </figure>
  240. Similarly, in <xref target="Tran01"/> the coding gain of the TDLT using fast
  241. factorizations with real coefficients produced by unconstrained optimization
  242. are
  243. <figure align="center">
  244. <artwork align="center"><![CDATA[
  245. 4x8 8x16 16x32
  246. +-----------+-----------+-----------+
  247. Type III TDLT | 8.6349 dB | 9.6115 dB | 9.9496 dB |
  248. Type IV TDLT | 8.6349 dB | 9.6005 dB | 9.9057 dB |
  249. +-----------+-----------+-----------+
  250. ]]></artwork>
  251. </figure>
  252. </t>
  253. </section>
  254. <!--section anchor="sba" title="Side-Band Attenuation">
  255. </section-->
  256. <section anchor="transform_width" title="Transform Width">
  257. <t>
  258. In general, the wider the transform, the higher the coding gain: a 16-point
  259. DCT will always have a higher coding gain than a 4-point DCT.
  260. In the case of lapped transform, the width of the transform is more than just
  261. counting the number of points, it involves the shape of the basis functions.
  262. At equal coding gain, a narrower transform is better because it causes a
  263. smaller amount of ringing around edges.
  264. We define the width of the transform as
  265. <figure align="center">
  266. <artwork align="center"><![CDATA[
  267. 1/4
  268. / sum_ij ( H[i,j]^2 * (j-N+1/2)^4 ) \
  269. w = C * | ---------------------------------- | ,
  270. \ sum_ij ( H[i,j]^2 ) /
  271. ]]></artwork>
  272. </figure>
  273. where C=2.991 is a constant calibrated such that the width of the 1024-point
  274. non-overlapped DCT is equal to 1024.
  275. </t>
  276. </section>
  277. <!--section anchor="orthogonality" title="Orthogonality">
  278. </section-->
  279. </section>
  280. <section anchor="search" title="Optimal Transform Coefficients">
  281. <t>
  282. Of the four metrics described in <xref target="metrics"/> we chose to optimize
  283. our transform parameters for the highest coding gain.
  284. </t>
  285. <t>
  286. To avoid the use of floating point operations, we use dyadic rationals to
  287. represent the parameters of our TDLT.
  288. These are the p's, q's and s's that describe the V matrix in the pre-filter.
  289. We chose a base of 2^6 because it offered enough resolution to find good
  290. approximations of the optimal values for the p's, q's, and s's and still
  291. allowed us to fit the results of multiplications in a 16 bit word.
  292. Increasing the base to 2^8 improves the achievable coding gain of the 4x8
  293. transform by less than 0.002&nbsp;dB.
  294. On the other hand, dropping it even one bit to 2^5 lowers the coding gain by
  295. 0.037&nbsp;dB.
  296. </t>
  297. <section anchor="brute" title="Exhaustive Search">
  298. <t>
  299. For the smaller lapped transforms, it is possible to simply do an exhaustive
  300. search and check all possible transform candidates to find the one with the
  301. best coding gain.
  302. The limitation that the p's, q's, and s's all be dyadic rationals allows us to
  303. simply enumerate all reasonable values.
  304. Additional constraints allowed us to further reduce the search space.
  305. Because the p's and q's are liftings steps that represent rotations in the plane
  306. their, values are between -1.0 and 1.0.
  307. Likewise the limitation that the pre- and post-filter steps be reversible
  308. requires that the scale factors be &gte; 1.0, otherwise information would be
  309. lost during the transform.
  310. Finally, all things equal we prefer smaller scale factors as it makes quantizing
  311. and encoding the coefficients cheaper.
  312. We thus cap the scale factors at 2.0.
  313. Based on some limited experimentation, scale factors larger than this do not
  314. appear to produce useful transforms according to our metrics, anyway.
  315. </t>
  316. <t>
  317. With a dyadic rational base of 2^6, the number of possible candidates to
  318. consider is
  319. <figure align="left">
  320. <artwork align="left"><![CDATA[
  321. |C| = (2*(2^6)+1)^(|p|+|q|) * (2^6+1)^|s|
  322. = (2*(2^6)+1)^(2*(N/2-1)) * (2^6+1)^(N/2)
  323. ]]></artwork>
  324. </figure>
  325. Thus for the transform sizes we are interested in, the number of candidates is
  326. tractable only for the 4x8 case:
  327. <figure align="left">
  328. <artwork align="left"><![CDATA[
  329. N |C|
  330. +-----+------------------+
  331. 4x8 TDLT | 4 | 68161536 |
  332. 8x16 TDLT | 8 | 7.731400 * 10^19 |
  333. 16x32 TDLT | 16 | 9.947082 * 10^43 |
  334. +-----+------------------+
  335. ]]></artwork>
  336. </figure>
  337. </t>
  338. <t>
  339. An exhaustive search for parameters that give the optimal coding gain for the
  340. 4x8 TDLT are below:
  341. <figure align="left">
  342. <artwork align="left"><![CDATA[
  343. +-----+--------+ +-----+--------+ +-----+--------+
  344. | p_0 | -11/64 | | q_0 | 36/64 | | s_0 | 91/64 |
  345. +-----+--------+ +-----+--------+ | s_1 | 85/64 |
  346. +-----+--------+
  347. ]]></artwork>
  348. </figure>
  349. </t>
  350. </section>
  351. <section anchor="stochastic" title="Stochastic Search">
  352. <t>
  353. For the larger lapped transforms, doing an exhaustive search is not possible.
  354. Instead we formulate the optimization problem as an integer programming problem
  355. and use a robust industrial solver to find optimal integer values for the
  356. p's, q's, and s's.
  357. </t>
  358. <t>
  359. For the 8x16 TDLT, the parameters are below:
  360. <figure align="left">
  361. <artwork align="left"><![CDATA[
  362. +-----+--------+ +-----+--------+ +-----+--------+
  363. | p_0 | -23/64 | | q_0 | 48/64 | | s_0 | 90/64 |
  364. | p_1 | -18/64 | | q_1 | 34/64 | | s_1 | 73/64 |
  365. | p_2 | -6/64 | | q_2 | 20/64 | | s_2 | 72/64 |
  366. +-----+--------+ +-----+--------+ | s_3 | 75/64 |
  367. +-----+--------+
  368. ]]></artwork>
  369. </figure>
  370. </t>
  371. <t>
  372. For the 16x32 TDLT, the parameters are below:
  373. <figure align="left">
  374. <artwork align="left"><![CDATA[
  375. +-----+--------+ +-----+--------+ +-----+--------+
  376. | p_0 | -24/64 | | q_0 | 50/64 | | s_0 | 90/64 |
  377. | p_1 | -23/64 | | q_1 | 40/64 | | s_1 | 74/64 |
  378. | p_2 | -17/64 | | q_2 | 31/64 | | s_2 | 73/64 |
  379. | p_3 | -12/64 | | q_3 | 22/64 | | s_3 | 71/64 |
  380. | p_4 | -14/64 | | q_4 | 18/64 | | s_4 | 67/64 |
  381. | p_5 | -13/64 | | q_5 | 16/64 | | s_5 | 67/64 |
  382. | p_6 | -7/64 | | q_6 | 11/64 | | s_6 | 67/64 |
  383. +-----+--------+ +-----+--------+ | s_7 | 72/64 |
  384. +-----+--------+
  385. ]]></artwork>
  386. </figure>
  387. </t>
  388. <t>
  389. In order to confirm that the integer approximations found are in fact optimal,
  390. we can compare them with the optimal real valued coding gains for the three
  391. lapped-transforms we are proposing.
  392. In <xref target="Tran01"/> a numeric solver was used to find optimal values for
  393. a Type&nbsp;IV lapped transform.
  394. <figure align="left">
  395. <artwork align="left"><![CDATA[
  396. 4x8 8x16 16x32
  397. +------------+------------+------------+
  398. Real Valued | 8.6349 dB | 9.6005 dB | 9.9057 dB |
  399. Approximate | 8.63473 dB | 9.60021 dB | 9.89338 dB |
  400. +------------+------------+------------+
  401. Loss | 0.00017 dB | 0.00029 dB | 0.01232 dB |
  402. +------------+------------+------------+
  403. ]]></artwork>
  404. </figure>
  405. </t>
  406. </section>
  407. <section anchor="ramp" title="Ramp Constraint">
  408. <t>
  409. It is also possible to constrain the lapped transform so that it is
  410. (1,2)-regular&nbsp;<xref target="DT03"/>, i.e., that it has one vanishing
  411. moment in the analysis filter and two vanishing moments in the synthesis
  412. filter.
  413. This allows the synthesis filter to reconstruct any piecewise linear function
  414. solely from the DC coefficients.
  415. This causes the shape of the DC basis function to be a symmetric linear ramp.
  416. This can be particularly useful when it matches the shape of other windowing
  417. functions used in the codec.
  418. For example, a linear window is commonly used with Overlapped Block Motion
  419. Compensation (OBMC), which is one possible approach for avoiding blocking
  420. artifacts in the motion-compensation stage of the codec.
  421. More vanishing moments are possible, allowing reconstruction of piecewise
  422. quadratic or even higher-order functions, but these require additional overlap
  423. stages.
  424. </t>
  425. <t>
  426. This regularity can be enforced solely by enforcing a series of constraints on
  427. the scale factors, s_i.
  428. <figure align="center">
  429. <artwork align="center"><![CDATA[
  430. s_0 = N*(1 - q_0)
  431. N / \
  432. s_i = ------- * | (q_{i-1} - 1)*p_{i-1} - q_i | , for i > 0
  433. 2*i + 1 \ /
  434. ]]></artwork>
  435. </figure>
  436. Since 2*i + 1 is odd, but we want s_i to be a dyadic rational value, the
  437. remainder of the expression must be evenly divisible by (2*i+1).
  438. A similar set of constraints can be derived for Type&nbsp;III, but they involve
  439. more of the p's and q's per s_i value, and thus have far fewer admissible
  440. solutions when coupled with the dyadic rational constraint.
  441. </t>
  442. <t>
  443. The additional restrictions described above greatly reduce the number of
  444. combinations to consider, both because there are fewer parameters (the s_i's
  445. can no longer be chosen independently) and because there are fewer
  446. combinations of parameter values which produce dyadic rational coefficients.
  447. With these constraints, the number of combinations is small enough that an
  448. exhaustive search is now tractable for the 8x16 TDLT.
  449. <figure align="left">
  450. <artwork align="left"><![CDATA[
  451. N |C|
  452. +-----+-----------+
  453. 4x8 TDLT | 4 | 442 |
  454. 8x16 TDLT | 8 | 331677320 |
  455. +-----+-----------+
  456. ]]></artwork>
  457. </figure>
  458. An exhaustive search for parameters that give the optimal coding gain under the
  459. ramp and dyadic rational constraints for the 4x8 and 8x16 TDLT are below:
  460. <figure align="left">
  461. <artwork align="left"><![CDATA[
  462. +-----+--------+ +-----+--------+ +-----+--------+
  463. | p_0 | -16/64 | | q_0 | 41/64 | | s_0 | 92/64 |
  464. +-----+--------+ +-----+--------+ | s_1 | 93/64 |
  465. +-----+--------+
  466. ]]></artwork>
  467. </figure>
  468. <figure align="left">
  469. <artwork align="left"><![CDATA[
  470. +-----+--------+ +-----+--------+ +-----+--------+
  471. | p_0 | -24/64 | | q_0 | 53/64 | | s_0 | 88/64 |
  472. | p_1 | -20/64 | | q_1 | 40/64 | | s_1 | 75/64 |
  473. | p_2 | -4/64 | | q_2 | 24/64 | | s_2 | 76/64 |
  474. +-----+--------+ +-----+--------+ | s_3 | 76/64 |
  475. +-----+--------+
  476. ]]></artwork>
  477. </figure>
  478. </t>
  479. <t>
  480. Unfortunately, in the 16x32 TDLT case the number of combinations is still not
  481. tractable, even with these additional constraints.
  482. Again, we use an integer programming model to solve for the integer parameters
  483. that optimize coding gain in this context.
  484. <figure align="left">
  485. <artwork align="left"><![CDATA[
  486. +-----+--------+ +-----+--------+ +-----+--------+
  487. | p_0 | -32/64 | | q_0 | 59/64 | | s_0 | 80/64 |
  488. | p_1 | -28/64 | | q_1 | 53/64 | | s_1 | 72/64 |
  489. | p_2 | -24/64 | | q_2 | 46/64 | | s_2 | 73/64 |
  490. | p_3 | -32/64 | | q_3 | 41/64 | | s_3 | 68/64 |
  491. | p_4 | -24/64 | | q_4 | 35/64 | | s_4 | 72/64 |
  492. | p_5 | -13/64 | | q_5 | 24/64 | | s_5 | 74/64 |
  493. | p_6 | -2/64 | | q_6 | 12/64 | | s_6 | 74/64 |
  494. +-----+--------+ +-----+--------+ | s_7 | 70/64 |
  495. +-----+--------+
  496. ]]></artwork>
  497. </figure>
  498. </t>
  499. <t>
  500. <figure align="left">
  501. <artwork align="left"><![CDATA[
  502. 4x8 8x16 16x32
  503. +------------+------------+------------+
  504. Dyadic | 8.63473 dB | 9.60021 dB | 9.89338 dB |
  505. Ramp + Dyadic | 8.59886 dB | 9.56161 dB | 9.78294 dB |
  506. +------------+------------+------------+
  507. Loss | 0.03587 dB | 0.0386 dB | 0.11044 dB |
  508. +------------+------------+------------+
  509. ]]></artwork>
  510. </figure>
  511. </t>
  512. </section>
  513. </section>
  514. <section anchor="future_work" title="Future Work">
  515. <t>
  516. There are a number of challenges that must be addressed to incorporate TDLTs
  517. into a codec.
  518. <list style="numbers">
  519. <t>
  520. Intra Prediction.
  521. Since the final pixel values of a block are not available until after the
  522. post-filter runs, they cannot be used to predict neighboring blocks.
  523. There are a number of possible solutions to this.
  524. For example, one could simply use pixels from outside the overlap region.
  525. However, as these pixels are farther away, they are poorer predictors, and the
  526. extra distance reduces the range of prediction directions which have enough
  527. neighbors available to form an adequate
  528. extrapolation&nbsp;<xref target="OP11"/>.
  529. An alternate approach is to perform the prediction in the frequency domain.
  530. Initial experiments suggest that this is just as effective as prediction in the
  531. time domain, and has similar computational requirements.
  532. </t>
  533. <t>
  534. Motion Compensation.
  535. There have been several lapped transform proposals that perform block-by-block
  536. motion compensation by simply expanding the size of the prediction region for
  537. each block&nbsp;<xref target="TT01"/>,&nbsp;<xref target="OPT11"/>.
  538. However, in addition to increasing the amount of motion-compensated prediction
  539. pixels that must be computed by a factor of four, this also increases the
  540. number of applications of the pre- and post-filter by a factor of four, since
  541. this must now be done separately for each block, using the motion-compensated
  542. frame difference for that block.
  543. An alternate approach is simply perform motion compensation of the frame in a
  544. completely separate step, prior to any transform, using any method desired.
  545. The lapping can then be applied to this motion-compensated prediction,
  546. producing per-block predictors.
  547. This still allows the prediction mode (inter, intra, bi-prediction, etc.) to be
  548. chosen on a block-by-block basis.
  549. It also interacts well with other techniques designed to operate in the
  550. frequency domain, such as the Pyramid Vector Quantization (PVQ) proposed
  551. elsewhere.
  552. The downside is that motion estimation in the encoder needs to be performed for
  553. regions slightly beyond the current block.
  554. However, this is already required by blocking-artifact-free motion compensation
  555. techniques, such as Overlapped Block Motion Compensation (OBMC).
  556. Experience with OBMC has shown that an encoder can mostly ignore look-ahead and
  557. still get acceptable results, unlike other techniques, such as control-grid
  558. interpolation (CGI).
  559. </t>
  560. <t>
  561. Multiple Block Sizes.
  562. Multiple block size support is important for lapped transforms, since the
  563. larger support region increases their susceptibility to ringing artifacts
  564. compared to a non-overlapped transform with the same number of coefficients
  565. (though it is greatly reduced compared to a non-overlapped transform with a
  566. support region of the same size).
  567. The simplest approach is to require that the size of the overlap filter be
  568. constrained by the smallest block adjacent to a given edge.
  569. This also requires some amount of look-ahead in the encoder, but in addition
  570. has an effect on the coding syntax, since this constraint must be communicated
  571. to the decoder in order for it to perform motion compensation.
  572. It is straightforward to make this information non-redundant, i.e., to use it
  573. to constraint the actual block sizes of future blocks when they are eventually
  574. decoded.
  575. </t>
  576. </list>
  577. </t>
  578. </section>
  579. <section title="IANA Considerations">
  580. <t>
  581. This document has no actions for IANA.
  582. </t>
  583. </section>
  584. <section title="Security Considerations">
  585. <t>
  586. This draft has no security considerations.
  587. </t>
  588. </section>
  589. <section anchor="Acknowledgments" title="Acknowledgments">
  590. <t>
  591. Thanks to Greg Maxwell and Jean-Marc Valin for their assistance in the
  592. experimentation and other valuable contributions to this document.
  593. </t>
  594. </section>
  595. </middle>
  596. <back>
  597. <!--references title="Normative References">
  598. <?rfc include="http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119.xml"?>
  599. </references-->
  600. <references title="Informative References">
  601. <reference anchor="DT03">
  602. <front>
  603. <title>Regularity-Constrained Pre- and Post-Filtering for Block DCT Based Systems</title>
  604. <author initials="W." surname="Dai" fullname="Wei Dai"/>
  605. <author initials="T.D." surname="Tran" fullname="Trac D. Tran"/>
  606. <date month="October" year="2003"/>
  607. </front>
  608. <seriesInfo name="IEEE Transactions on Signal Processing" value="51(10):2568--2581"/>
  609. </reference>
  610. <reference anchor="Malv89" target="http://research.microsoft.com/apps/pubs/default.aspx?id=102073">
  611. <front>
  612. <title>The LOT: Transform Coding Without Blocking Effects</title>
  613. <author initials="H.S." surname="Malvar"/>
  614. <author initials="D.H." surname="Staelin"/>
  615. <date month="April" year="1989"/>
  616. </front>
  617. <seriesInfo name="IEEE Transactions on Acoustics, Speech, and Signal Processing" value=""/>
  618. </reference>
  619. <reference anchor="OP11">
  620. <front>
  621. <title>Intra-Frame Prediction with Lapped Transforms for Image Coding</title>
  622. <author initials="R.G." surname="de Oliveria" fullname="Rafael G. de Oliveria"/>
  623. <author initials="B." surname="Pesquet-Popescu" fullname="Beatrice Pesquet-Popescu"/>
  624. <date month="May" year="2011"/>
  625. </front>
  626. <seriesInfo name="Proc. of the 36th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'11)" value="pp. 805--808"/>
  627. </reference>
  628. <reference anchor="OPT11">
  629. <front>
  630. <title>Inter Prediction Using Lapped Transforms for Advanced Video Coding</title>
  631. <author initials="R.G." surname="de Oliveria" fullname="Rafael G. de Oliveria"/>
  632. <author initials="B." surname="Pesquet-Popescu" fullname="Beatrice Pesquet-Popescu"/>
  633. <author initials="M." surname="Trocan" fullname="Maria Trocan"/>
  634. <date month="September" year="2011"/>
  635. </front>
  636. <seriesInfo name="Proc. of the 18th IEEE International Conference on Image Processing (ICIP'11)" value="pp. 3705--3708"/>
  637. </reference>
  638. <reference anchor="Tran01">
  639. <front>
  640. <title>Lapped Transform via Time-Domain Pre- and Post-Processing</title>
  641. <author initials="T.D." surname="Tran"/>
  642. <date month="October" year="2001"/>
  643. </front>
  644. <seriesInfo name="IEEE Transactions on Signal Processing" value=""/>
  645. </reference>
  646. <reference anchor="TT01">
  647. <front>
  648. <title>Lapped Transform Based Video Coding</title>
  649. <author initials="T.D." surname="Tran" fullname="Trac D. Tran"/>
  650. <author initials="C." surname="Tu" fullname="Chengjie Tu"/>
  651. <date month="July" year="2001"/>
  652. </front>
  653. <seriesInfo name="Proc. of the 24th SPIE Conference on Applications of Digital Image Processing" value="vol. 4472, pp. 319--333"/>
  654. </reference>
  655. <reference anchor="Terr12">
  656. <front>
  657. <title>Introduction to Video Coding Part 1: Transform Coding</title>
  658. <author initials="T.B." surname="Terriberry"/>
  659. <date month="February" year="2012"/>
  660. </front>
  661. </reference>
  662. </references>
  663. </back>
  664. </rfc>