This is likely to be a bandwidth issue between the BT817 and the SPI-Flash.
I converted NotoSansSC-Regular.otf with your table and EAB outputs 48x48 pixel for the glyhs for a size of 36.
So the first thing you can try is to use much smaller fonts, like 24.
Just to test this.
24 is converted here to 24x32 pixel and is using ASTC 8x8 on setting "auto".
If that works with more characters this is a bandwidth issue.
Next you can setup the font-cache for the larger font, see CMD_FONTCACHE.
This way a subset of glyphs is fetched from RAM_G.
CMD_FONTCACHEQUERY helps tuning the memory for the cache.
ASTC 10x5 is 2.56 bits per pixel.
So 40 * 40 * 2.56 is 4096.
A glyph needs to be 32 byte aligned but 4096 is divisible by 32.
For 20 chars fontcache we need 4096 * 20 = 81920 bytes.
Note that the start of the cache needs to be 64-byte aligned.
ASTC 8x8 uses 2 bits per pixel:
40 * 40 * 2 = 3200.
20 * 3200 = 64000 bytes
EAB 2.6.1 converts NotoSansSC-Regular.otf to ASTC 8x8 for me when set to "Auto".
Generated Folder: NotoSansSC-Regular_36_Extend
Format: COMPRESSED_RGBA_ASTC_8x8_KHR
Compressed: thorough
Layout Width: 80
Layout Height: 8
Pixel Width: 40
Pixel Height: 64
Number of characters in xfont file: 65408
Number of characters in user input file: 292
Success: 287
Fail: 5
Fail code point: 0x5f00, 0x8a00, 0x9000, 0x9600, 0x9700
NotoSansSC-Regular_36_ASTC.xfont and NotoSansSC-Regular_36_ASTC.glyph are generated successfully.
40 * 64 * 2 = 5120
20 * 5120 = 102400 bytes
The "Fail: 5" is interesting as EAB 2.5.1 did not have any fails.
Hmm, when I manually set it to 10x5 I get this:
Generated Folder: NotoSansSC-Regular_36_Extend
Format: COMPRESSED_RGBA_ASTC_10x5_KHR
Compressed: thorough
Layout Width: 64
Layout Height: 8
Pixel Width: 40
Pixel Height: 40
Number of characters in xfont file: 65408
Number of characters in user input file: 292
Success: 287
Fail: 5
Fail code point: 0x5f00, 0x8a00, 0x9000, 0x9600, 0x9700
So the same amount of characters for the same map but somehow the glyphs are smaller?
How? Why?
Ok, setting it to 10x8 for 1.6 bits per pixel the result is 40x40 pixel for the glyphs.
40 * 40 * 1.6 = 2560 (2560 % 32 = 0)
20 * 2560 = 51200 bytes
And one last thing.
Setup REG_AH_HCYCLE_MAX to for example HCYCLE + 500, the reset value is zero.