I have used N=200 random files from the LibriSpeech corpus to run a comparison between Codec2 and Codec2-mod. I used ViSQOL perceptual quality estimator to calculate the Mean Opinion Score (MOS) for original-decompressed pairs for both implementations. Results are shown in the table at the bottom of this post(copied verbatim from terminal).
The few mismatches are most likely caused by slightly different floating point maths (comparisons and fast cosine and arc cosine functions). Those may diverge slightly given edge-case input signal vectors. Still, Codec2-mod is functionally bit-exact with reference Codec2 for almost all inputs.
The +0.006 difference in MOS is not significant (and even imperceptible). This number does not signify any perceptual improvement. I’m pretty sure that the value would converge to 0.0, given enough samples. A close to zero value means that there is no regression (which is good!).
This is the histogram for both implementations:
“Bit-diff” tells how many bytes were different in the encoded (compressed) data.
Note: even a single bit flip still counts as a 1 byte difference.
Terminal dump for 200 samples:
File | MOS-ref | MOS-mod | dMOS | Bit-diff -----------------------------------------+----------+----------+----------+--------- 2412-153948-0003.wav | 2.671 | 2.753 | 0.082 | - 7976-105575-0013.wav | 2.485 | 2.519 | 0.034 | - 3752-4944-0005.wav | 2.725 | 2.815 | 0.090 | - 2277-149896-0009.wav | 3.622 | 3.583 | -0.039 | - 2412-153948-0005.wav | 2.165 | 2.371 | 0.206 | - 3752-4944-0034.wav | 2.605 | 2.601 | -0.004 | - 5895-34622-0002.wav | 1.137 | 1.376 | 0.239 | - 3576-138058-0020.wav | 2.454 | 2.503 | 0.049 | 1 6295-64301-0004.wav | 2.578 | 2.591 | 0.013 | - 3576-138058-0022.wav | 1.740 | 1.925 | 0.185 | - 2277-149896-0028.wav | 2.790 | 2.810 | 0.020 | - 6313-66129-0023.wav | 1.668 | 1.628 | -0.040 | - 5694-64025-0004.wav | 2.558 | 2.596 | 0.037 | - 6295-244435-0018.wav | 2.390 | 2.398 | 0.008 | - 5895-34615-0016.wav | 1.164 | 1.053 | -0.111 | - 1272-135031-0018.wav | 1.582 | 1.631 | 0.049 | - 6241-61946-0020.wav | 2.437 | 2.467 | 0.030 | - 652-130737-0007.wav | 2.208 | 2.186 | -0.021 | - 251-137823-0019.wav | 2.602 | 2.627 | 0.025 | - 2428-83705-0007.wav | 2.444 | 2.449 | 0.005 | - 251-137823-0009.wav | 2.286 | 2.348 | 0.062 | - 5338-284437-0003.wav | 2.711 | 2.713 | 0.002 | - 3853-163249-0016.wav | 3.261 | 3.290 | 0.029 | - 6345-93306-0007.wav | 2.561 | 2.462 | -0.099 | - 2277-149896-0011.wav | 3.570 | 3.571 | 0.001 | - 777-126732-0040.wav | 2.928 | 2.956 | 0.028 | - 7850-111771-0001.wav | 2.903 | 2.916 | 0.014 | - 251-137823-0025.wav | 1.848 | 1.754 | -0.094 | - 777-126732-0043.wav | 3.485 | 3.473 | -0.012 | - 1993-147149-0004.wav | 2.336 | 2.318 | -0.018 | - 251-118436-0003.wav | 2.251 | 2.296 | 0.046 | - 2412-153954-0004.wav | 2.144 | 2.095 | -0.050 | - 3536-8226-0029.wav | 2.573 | 2.574 | 0.001 | - 6319-275224-0010.wav | 1.893 | 1.980 | 0.087 | - 6295-244435-0038.wav | 2.543 | 2.538 | -0.005 | - 3853-163249-0042.wav | 2.771 | 2.830 | 0.060 | - 2078-142845-0017.wav | 1.708 | 1.721 | 0.013 | - 3536-8226-0008.wav | 2.859 | 2.890 | 0.031 | - 2035-147961-0040.wav | 2.628 | 2.632 | 0.004 | - 8297-275155-0002.wav | 2.493 | 2.553 | 0.061 | - 652-130726-0032.wav | 3.322 | 3.411 | 0.089 | - 6345-93306-0013.wav | 2.649 | 2.653 | 0.004 | - 422-122949-0026.wav | 2.961 | 2.964 | 0.002 | - 6313-66129-0033.wav | 1.186 | 1.220 | 0.035 | - 3081-166546-0014.wav | 2.014 | 2.027 | 0.013 | - 251-118436-0006.wav | 2.051 | 2.122 | 0.071 | 1 7976-105575-0021.wav | 2.775 | 2.801 | 0.026 | - 5536-43359-0016.wav | 2.288 | 2.178 | -0.110 | - 5895-34615-0001.wav | 1.000 | 1.000 | 0.000 | - 3752-4944-0021.wav | 2.455 | 2.525 | 0.070 | - 2277-149897-0010.wav | 2.445 | 2.446 | 0.001 | - 1988-24833-0005.wav | 1.729 | 1.838 | 0.109 | - 422-122949-0031.wav | 2.849 | 2.837 | -0.012 | - 2035-147961-0022.wav | 1.972 | 1.976 | 0.004 | - 6241-61943-0008.wav | 1.492 | 1.562 | 0.070 | - 6313-76958-0028.wav | 1.730 | 1.797 | 0.067 | - 8297-275155-0004.wav | 2.343 | 2.348 | 0.005 | - 2412-153954-0013.wav | 2.108 | 2.107 | -0.001 | - 2035-147961-0015.wav | 2.188 | 2.127 | -0.061 | - 8842-302201-0015.wav | 1.529 | 1.567 | 0.038 | - 8842-302201-0004.wav | 2.199 | 2.176 | -0.023 | - 2035-147961-0029.wav | 2.153 | 2.142 | -0.011 | - 422-122949-0013.wav | 3.050 | 3.022 | -0.028 | - 3853-163249-0050.wav | 2.504 | 2.505 | 0.001 | - 6345-93302-0018.wav | 3.504 | 3.494 | -0.010 | - 652-129742-0002.wav | 2.472 | 2.692 | 0.219 | - 2803-161169-0005.wav | 2.410 | 2.581 | 0.172 | 1 1993-147965-0000.wav | 2.165 | 2.143 | -0.022 | - 3853-163249-0032.wav | 3.155 | 3.135 | -0.020 | - 7976-110523-0013.wav | 2.722 | 2.713 | -0.010 | - 8842-302201-0013.wav | 2.133 | 2.014 | -0.119 | - 5536-43359-0005.wav | 1.916 | 1.950 | 0.034 | - 1462-170142-0015.wav | 2.648 | 2.642 | -0.006 | - 6295-244435-0011.wav | 2.189 | 2.190 | 0.001 | - 84-121123-0025.wav | 1.626 | 1.611 | -0.015 | - 7850-286674-0016.wav | 2.700 | 2.694 | -0.006 | - 5536-43359-0000.wav | 2.352 | 2.297 | -0.056 | - 5895-34629-0001.wav | 2.356 | 2.195 | -0.160 | - 1988-148538-0015.wav | 2.703 | 2.618 | -0.085 | - 2803-154328-0002.wav | 2.661 | 2.734 | 0.074 | - 3752-4943-0011.wav | 2.782 | 2.858 | 0.076 | - 1993-147149-0026.wav | 3.070 | 3.105 | 0.035 | - 2412-153947-0014.wav | 2.362 | 2.269 | -0.094 | - 6345-64257-0020.wav | 1.477 | 1.468 | -0.008 | - 3576-138058-0037.wav | 2.578 | 2.606 | 0.028 | - 251-118436-0000.wav | 2.547 | 2.582 | 0.035 | - 5895-34615-0006.wav | 1.751 | 1.736 | -0.015 | - 652-130737-0004.wav | 2.253 | 2.265 | 0.012 | - 84-121550-0027.wav | 1.954 | 1.896 | -0.058 | 8 6313-66125-0005.wav | 1.309 | 1.316 | 0.007 | - 6319-275224-0015.wav | 2.026 | 2.024 | -0.003 | - 5895-34622-0019.wav | 1.062 | 1.084 | 0.022 | - 3170-137482-0036.wav | 2.855 | 2.861 | 0.006 | - 251-136532-0008.wav | 2.300 | 2.314 | 0.014 | - 5694-64029-0010.wav | 2.638 | 2.660 | 0.022 | - 174-168635-0008.wav | 1.668 | 1.688 | 0.020 | - 2086-149220-0032.wav | 2.899 | 2.935 | 0.035 | - 2902-9008-0010.wav | 2.062 | 1.986 | -0.077 | - 3081-166546-0043.wav | 2.198 | 2.249 | 0.051 | - 3853-163249-0001.wav | 3.033 | 2.998 | -0.035 | - 777-126732-0048.wav | 2.165 | 2.207 | 0.043 | - 2277-149874-0002.wav | 3.009 | 3.015 | 0.006 | - 6345-93306-0009.wav | 2.264 | 2.246 | -0.018 | - 8297-275154-0003.wav | 2.524 | 2.513 | -0.011 | - 3536-23268-0001.wav | 2.754 | 2.797 | 0.043 | - 5895-34615-0004.wav | 1.662 | 1.634 | -0.028 | - 5536-43363-0002.wav | 2.288 | 2.276 | -0.012 | - 6319-64726-0018.wav | 3.163 | 3.199 | 0.036 | - 777-126732-0075.wav | 2.281 | 2.285 | 0.004 | - 84-121123-0012.wav | 2.777 | 2.752 | -0.025 | - 2277-149896-0007.wav | 3.321 | 3.295 | -0.025 | - 251-137823-0015.wav | 3.207 | 3.213 | 0.007 | - 7976-110124-0016.wav | 2.647 | 2.642 | -0.005 | - 174-84280-0012.wav | 2.096 | 2.179 | 0.083 | - 6241-61943-0005.wav | 2.761 | 2.762 | 0.001 | - 1919-142785-0035.wav | 1.371 | 1.366 | -0.005 | - 6345-64257-0016.wav | 2.503 | 2.528 | 0.025 | - 3081-166546-0083.wav | 2.841 | 2.757 | -0.084 | - 84-121550-0025.wav | 2.127 | 2.140 | 0.012 | 14 2277-149897-0026.wav | 3.443 | 3.487 | 0.044 | - 84-121550-0022.wav | 2.199 | 2.217 | 0.018 | - 1919-142785-0060.wav | 1.940 | 1.934 | -0.006 | - 1462-170138-0027.wav | 2.489 | 2.280 | -0.208 | - 3752-4944-0055.wav | 2.816 | 2.837 | 0.021 | - 6295-244435-0008.wav | 2.543 | 2.543 | 0.001 | - 6313-76958-0019.wav | 1.485 | 1.485 | 0.001 | - 174-50561-0002.wav | 2.786 | 2.808 | 0.022 | - 777-126732-0015.wav | 2.338 | 2.331 | -0.007 | - 2803-154328-0023.wav | 2.523 | 2.596 | 0.073 | - 2412-153954-0006.wav | 2.441 | 2.444 | 0.004 | - 6295-64301-0029.wav | 2.570 | 2.518 | -0.052 | - 251-137823-0006.wav | 2.715 | 2.749 | 0.033 | - 2035-147960-0016.wav | 2.726 | 2.603 | -0.123 | - 6319-57405-0003.wav | 2.387 | 2.332 | -0.055 | - 8297-275154-0007.wav | 2.626 | 2.703 | 0.077 | - 2078-142845-0002.wav | 2.094 | 2.098 | 0.004 | - 2277-149897-0019.wav | 1.261 | 1.230 | -0.031 | - 251-137823-0014.wav | 2.405 | 2.388 | -0.017 | - 777-126732-0066.wav | 2.684 | 2.730 | 0.045 | - 7976-105575-0012.wav | 3.016 | 2.958 | -0.058 | - 2277-149874-0006.wav | 2.962 | 3.030 | 0.068 | - 1988-24833-0007.wav | 1.620 | 1.692 | 0.072 | - 7850-73752-0012.wav | 2.905 | 2.922 | 0.017 | - 6313-76958-0001.wav | 1.618 | 1.640 | 0.022 | - 3752-4944-0022.wav | 2.456 | 2.440 | -0.016 | - 5895-34622-0004.wav | 1.171 | 1.169 | -0.002 | - 422-122949-0020.wav | 2.427 | 2.482 | 0.055 | - 3576-138058-0001.wav | 2.293 | 2.329 | 0.037 | - 2412-153954-0007.wav | 2.660 | 2.658 | -0.002 | - 422-122949-0034.wav | 2.713 | 2.714 | 0.001 | - 174-168635-0016.wav | 1.979 | 1.986 | 0.007 | - 5694-64038-0009.wav | 2.858 | 2.850 | -0.008 | - 2035-152373-0005.wav | 2.555 | 2.554 | -0.000 | - 6295-244435-0028.wav | 1.900 | 1.901 | 0.001 | - 2086-149214-0002.wav | 2.758 | 2.752 | -0.006 | - 2803-154320-0010.wav | 2.648 | 2.675 | 0.028 | - 174-168635-0021.wav | 2.020 | 2.092 | 0.072 | - 1462-170145-0003.wav | 2.380 | 2.363 | -0.017 | - 1462-170145-0004.wav | 2.249 | 2.266 | 0.018 | - 2035-147961-0026.wav | 2.802 | 2.775 | -0.027 | - 5895-34629-0014.wav | 2.673 | 2.670 | -0.003 | - 2428-83699-0004.wav | 2.967 | 2.960 | -0.007 | - 2428-83699-0008.wav | 2.682 | 2.673 | -0.010 | - 3536-8226-0018.wav | 2.465 | 2.447 | -0.018 | - 2803-154328-0007.wav | 2.059 | 2.068 | 0.008 | - 652-130726-0014.wav | 3.445 | 3.446 | 0.001 | - 174-50561-0015.wav | 2.060 | 1.934 | -0.126 | - 6319-64726-0008.wav | 2.660 | 2.739 | 0.079 | - 1462-170142-0041.wav | 3.108 | 3.105 | -0.002 | - 1993-147964-0002.wav | 1.977 | 1.901 | -0.076 | - 1919-142785-0020.wav | 2.416 | 2.319 | -0.097 | - 3081-166546-0060.wav | 2.437 | 2.588 | 0.151 | - 6345-93306-0005.wav | 1.684 | 1.680 | -0.004 | - 5694-64029-0020.wav | 2.499 | 2.516 | 0.018 | 2 6319-64726-0013.wav | 2.881 | 2.875 | -0.006 | - 6313-76958-0022.wav | 1.714 | 1.647 | -0.067 | - 3853-163249-0035.wav | 3.072 | 3.057 | -0.015 | - 3752-4943-0007.wav | 2.414 | 2.389 | -0.025 | - 3536-23268-0003.wav | 2.915 | 2.891 | -0.025 | - 174-168635-0003.wav | 1.648 | 1.598 | -0.049 | - 8842-304647-0013.wav | 1.870 | 1.868 | -0.002 | - 3752-4944-0009.wav | 2.641 | 2.536 | -0.105 | - 8297-275154-0016.wav | 2.502 | 2.458 | -0.044 | - 3081-166546-0003.wav | 2.397 | 2.415 | 0.018 | - 8297-275154-0013.wav | 2.761 | 2.758 | -0.003 | - 2086-149220-0023.wav | 3.113 | 3.069 | -0.044 | - 8297-275155-0026.wav | 2.467 | 2.525 | 0.058 | - 652-129742-0001.wav | 1.983 | 2.013 | 0.030 | - 2078-142845-0042.wav | 2.011 | 2.010 | -0.000 | - 174-50561-0013.wav | 2.335 | 2.308 | -0.027 | - 251-118436-0011.wav | 2.704 | 2.695 | -0.009 | - 1988-24833-0020.wav | 2.063 | 2.057 | -0.006 | - 5338-284437-0008.wav | 2.632 | 2.670 | 0.038 | - 1272-135031-0006.wav | 2.237 | 2.187 | -0.050 | - 3853-163249-0048.wav | 3.194 | 3.169 | -0.025 | - 1993-147149-0020.wav | 2.039 | 2.044 | 0.006 | - 1919-142785-0061.wav | 2.236 | 2.260 | 0.024 | - 1988-147956-0025.wav | 2.272 | 2.271 | -0.001 | - 2086-149220-0034.wav | 2.959 | 2.953 | -0.005 | - 3000-15664-0000.wav | 2.584 | 2.580 | -0.004 | - -----------------------------------------+----------+----------+----------+--------- MEAN | 2.402 | 2.407 | 0.006 | Processed OK : 200 Skipped (fail) : 0 Requested : 200
Leave a Reply