I've discovered a thread_queue_size number that seems to work on my system for this use-case but I'll test more (or repeat tests) to help the cause if someone can suggest something. Please advise. Thanks, Mark.

---

thread_queue_size 24883200 (to avoid "Thread message queue blocking")

The command line is VapourSynth (VSPIPE) piping raw frames plus FFMPEG deriving audio & subs from a concat of the VOBs, thus:

VSPIPE --y4m C:\AVout\VTS_04_1(210820-023517.49).mkv.vpy - | FFMPEG -thread_queue_size 24883200 -i pipe: -i "concat:h:\VIDEO_TS\VTS_04_1.VOB|h:\VIDEO_TS\VTS_04_2.VOB|h:\VIDEO_TS\VTS_04_3.VOB|h:\VIDEO_TS\VTS_04_4.VOB|h:\VIDEO_TS\VTS_04_5.VOB|h:\VIDEO_TS\VTS_04_6.VOB" -filter_complex "[0:v]settb=expr=1/120,setpts=expr=N,setdar=16/9,setsar=32/27[v];[1:a:0]atempo=1.001[a0];[1:a:1]atempo=1.001[a1];[1:a:2]atempo=1.001[a2];[1:a:3]atempo=1.001[a3];[1:a:4]atempo=1.001[a4]" -map "[v]" -map "[a0]" -map "[a1]" -map "[a2]" -map "[a3]" -map "[a4]" -map 1:s -codec:v libx265 -x265-params "crf=18:qcomp=0.80:sao=no:strong-intra-smoothing=no" -codec:a:0 ac3 -codec:a:1 ac3 -codec:a:2 ac3 -codec:a:3 ac3 -codec:a:4 ac3 -codec:s copy -r 120 C:\AVout\VTS_04_1(210820-023517.49).mkv

The result:

1, Transcoding proceeded at about speed=0.5.

2, While it ran, VSPIPE dominated the 8 threads (by a factor of ~2x over FFMPEG) and "System Commit" (sum of used RAM + used paging) maxed out at 126.3GB after rising steadily.

3, Console output of note was "Error: fwrite() call failed when writing frame: 331449, plane 2, errno: 22". It's hard to judge for sure, but it certainly appeared that the failure message printed at the same moment that "System Commit" topped out. On playback, video freezes at that precise point: frame 331449 (46:02.072) -- there should be 939036 frames (2:10:25.305).

---

thread_queue_size 2048

The command line is, again, VapourSynth (VSPIPE) piping raw frames et cetera (but this time with "-thread_queue_size 2048") thus:

VSPIPE --y4m C:\AVout\VTS_04_1(210820-042617.65).mkv.vpy - | FFMPEG -thread_queue_size 2048 -i pipe: -i "concat:h:\VIDEO_TS\VTS_04_1.VOB|h:\VIDEO_TS\VTS_04_2.VOB|h:\VIDEO_TS\VTS_04_3.VOB|h:\VIDEO_TS\VTS_04_4.VOB|h:\VIDEO_TS\VTS_04_5.VOB|h:\VIDEO_TS\VTS_04_6.VOB" -filter_complex "[0:v]settb=expr=1/120,setpts=expr=N,setdar=16/9,setsar=32/27[v];[1:a:0]atempo=1.001[a0];[1:a:1]atempo=1.001[a1];[1:a:2]atempo=1.001[a2];[1:a:3]atempo=1.001[a3];[1:a:4]atempo=1.001[a4]" -map "[v]" -map "[a0]" -map "[a1]" -map "[a2]" -map "[a3]" -map "[a4]" -map 1:s -codec:v libx265 -x265-params "crf=18:qcomp=0.80:sao=no:strong-intra-smoothing=no" -codec:a:0 ac3 -codec:a:1 ac3 -codec:a:2 ac3 -codec:a:3 ac3 -codec:a:4 ac3 -codec:s copy -r 120 C:\AVout\VTS_04_1(210820-042617.65).mkv

The result:

1, Transcoding proceeded at about speed=1.0 with "Thread message queue blocking; consider raising the thread_queue_size option (current value: 2048)".

2, While VSPIPE ran, it was FFMPEG (not VSPIPE) that dominated (by a factor of ~5x over VSPIPE) and "System Commit" rose, then leveled out at only 5.9GB.

3, All is well with playback.

---

Further Runs

I halved the bad thread_queue_size to 12441600, then halved it again to 6220800, et cetera, while monitoring "System Commit" in each run. With each run, as I lowered thread_queue_size, I saw "System Commit" and CPU utilization beginning to balance out between VSPIPE and FFMPEG.

---

Outcome

At "-thread_queue_size 48600", I once again began getting "Thread message queue blocking; consider raising the thread_queue_size option (current value: 48600)" and things settled down: FFMPEG & VSPIPE reversed dominance over CPU utilization (with FFMPEG now dominating) and "System Commit" rose linearly to 28.5GB and then leveled out.

It appears to me that "Thread message queue blocking" is a good thing -- a pipe that's not going to overflow if (or when) it runs for a long time. Is there a better way to assure this other than by twiddling thread_queue_size?

One other thing I don't understand: Why do I continue to get "Starting new cluster due to timestamp" notices -- I presume that's from the encoder -- when it's me that's writing the timestamps (via "settb=expr=1/120,setpts=expr=N")?

Notes: All code and files are on a solid-state RAID0, Windows10, 4 dual cores (8 threads), 3.6 GHz, 32GB RAM + 100GiB pagefile.sys. Tool used to monitor CPU utilization and "System Commit": Sysinternals, Process Explorer v16.21.

1 Answer

From

-thread_queue_size size (input)
This option sets the maximum number of queued packets when reading from the file or device. With low latency / high rate live streams, packets may be discarded if they are not read in a timely manner; setting this value can force ffmpeg to use a separate input thread and read packets as soon as they arrive. By default ffmpeg only do this if multiple inputs are specified.

Another clue is

 While it ran, VSPIPE dominated the 8 threads (by a factor of ~2x over FFMPEG) and "System Commit" (sum of used RAM + used paging) maxed out at 126.3GB after rising steadily.

This is a buffering issue - ffmpeg is unable to process frames as fast as they are received. We can try 3 solutions.

a. Write to a different SSD/HDD as the input file SSD/HDD. This will exclude write speeds as a failure parameter.

b. Sine you are using windows - start your process as

start /b /high "cmd"

c. There is a big difference in speed between named pipes vs using anonymous pipes in Windows.

From -

windows standard input is very slow. So I switched to named pipes. It's up to 10x faster then standart input (pipe:).

Replace the anonymous pipes inputs - and -i pipe:by

-i \\.\pipe\pipename out.mkv

Windows has no official cmd utility - you can use this -

13

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy