BRT Community

General Category => Discussion - EVE => Topic started by: Simon on March 06, 2020, 10:22:54 AM

Title: UTF-8 Encoding Error
Post by: Simon on March 06, 2020, 10:22:54 AM
Hi there,

I have an issue on the UTF-8 encoding used with the BT816-Chip. For the use of some special symbols i need font-elements that are adressed by UTF-8 encoding with four bytes. So I converted this font https://fontlibrary.org/de/font/symbola#Symbola-Regular (https://fontlibrary.org/de/font/symbola#Symbola-Regular) that contains the symbols I want to use with EVE Asset Builder v1.2.0.
After progammming the font on my flash, I tried to adress i.e. the UTF-8 code F09FA183 to type a special arrow. The display didn't show anything.
Then I tried to use a symbol with a three byte UTF-8 encoding i.e. E2AF86 (a different arrow symbol) and everything worked fine.

I also tried different ways to use the symbol:
One try was about putting each byte into a text array by hand like shown below, what worked as describes up to 3-byte encoding.

Code: [Select]
switch (direction)

         {
            case 0://arrow up
            {

               arrow_x = x1+((x2-x1)/2)-20;
               arrow_y = y1+((y2-y1)/2)-12;

               EVE_cmd_setfont2(13, SPECIAL_FONT2_RAM_ADR,0); //special font

         //this arrow is displayed correctly
               LCD_FuncData.text_buffer[0] =0xE2;
               LCD_FuncData.text_buffer[1] =0xAF;
               LCD_FuncData.text_buffer[2] = 0x85;
               LCD_FuncData.text_buffer[3] = 0;
               break;
            }
            case 1://arrow down
            {
               arrow_x = x1+((x2-x1)/2)-20;
               arrow_y = y1+((y2-y1)/2)-12;

               EVE_cmd_setfont2(13, SPECIAL_FONT2_RAM_ADR,0); //special font

         //this arrow is not shown on display
               LCD_FuncData.text_buffer[0] =0xF0;
               LCD_FuncData.text_buffer[1] =0x9F;
               LCD_FuncData.text_buffer[2] = 0xA1;
               LCD_FuncData.text_buffer[3] = 0x83;
               LCD_FuncData.text_buffer[4] = 0;
               break;
            }
  }

  if (button_down)
        {
               EVE_cmd_button(x1,y1,x2-x1,y2-y1,13,EVE_OPT_FLAT); //in here the button and its label is generated by sending the text array content to the BT-Chip
        }
        else
        {
               EVE_cmd_button(x1,y1,x2-x1,y2-y1,13,0);
        }

Another try I started in the EVE Screen Editor with only a few lines:

Code: [Select]
CLEAR(1, 1, 1)
CMD_SETFONT2(13, 0,0)
CMD_TEXT(123, 117, 13, 0, "🡃?")

I opened the font data in the emulator and loaded my font. This time the display showed two questionmarks instead of the arrow. In the Inspector view i could see that the generated code actually adresses the same section as for the questionmark character.

So my question is if there is an availability to use the Symbols with four byte UTF-8 encoding or if the BT-Chip restricts the usage of them.

Hopefully somebody can answer me that question.

Regards,
Simon
Title: Re: UTF-8 Encoding Error
Post by: BRT Community on March 09, 2020, 02:47:24 PM
Hello Simon,

I will pass this on to the R&D team for clarification on UTF-8 encoding support.

Best Regards,
BRT Community
Title: Re: UTF-8 Encoding Error
Post by: BRT Community on March 16, 2020, 11:29:55 AM
Hello Simon,

We discovered an issue with EAB regarding 4-byte UTF-8 characters.
EAB v1.4 is scheduled at Mar 25 and will include a fix for this.

Best Regards,
BRT Community
Title: Re: UTF-8 Encoding Error
Post by: Simon on March 16, 2020, 02:44:54 PM
Hello BRT Community,

many thanks for your reply. I am very excited about the revised version and would like to thank you for the bug fix.

Regards,
Simon
Title: Re: UTF-8 Encoding Error
Post by: Rudolph on April 09, 2020, 10:16:43 AM
As the 25th of march has passed now, any ETA for the new EAB?
I would not mind giving a beta a spin if it shows up in the "Test and Review Area".
Title: Re: UTF-8 Encoding Error
Post by: BRT Community on April 09, 2020, 01:58:54 PM
Hello,

I will chase the web team up to see what's what regarding the latest release of EAB.

Best Regards,
BRT Community
Title: Re: UTF-8 Encoding Error
Post by: Simon on April 28, 2020, 10:15:57 AM
Hello,
I have noticed that the new version of EVE Asset Builder is available and I am happy to see that the UTF8 problem is fixed. I have been testing version 1.4 since yesterday and still have the same problem in my test scenarios as before.
Could you send me a flash file and an ESE example with which the fix works? Or is there a possibility that someone from the R&D team contacts me?

I look forward to your feedback.
Simon
Title: Re: UTF-8 Encoding Error
Post by: BRT Community on April 28, 2020, 04:13:59 PM
Hello,

Just to clarify, after using the latest version of EAB, your special symbols are still not showing on screen when you attempt to use them in your code?

Best Regards,
BRT Community
Title: Re: UTF-8 Encoding Error
Post by: Simon on April 29, 2020, 07:26:33 AM
Yes exactly. With version 1.4 my icons are still not displayed correctly.
It is noticeable that the version of the EVE Font Conversion Utility in EAB version 1.2 has the same version number (v0.1.6) as in EAB version 1.4. Maybe it has something to do with this? Another thing that would suggest is that according to the release notes the compression for the UTF-8 code point setting has been fixed but does not seem to work either.

I would be glad about a feedback.

Regards Simon
Title: Re: UTF-8 Encoding Error
Post by: BRT Community on April 29, 2020, 11:58:02 AM
Hello

I have asked the development team if the font convertor version number should have incremented for the latest release.

In the meantime, we have given it a go here, please see below for the ESE source and attached screenshot:
Font_112.zip (ftp://u45902898-ide:Ftd1$erv@ftp.ftdichip.com/CES/EVE/Font_112.zip)

Best Regards,
BRT Community
Title: Re: UTF-8 Encoding Error
Post by: Rudolph on April 29, 2020, 06:11:36 PM
When I try to convert Symbola.ttf with EAB 1.4 to extended format, a size of 12 and "Full Unicode Characters" I get this in the "Result" window:
"unknown freetype error"
According to Fontrdop Symbola has 7956 glyphs.

When I switch to a NotoSans-Regular.ttf with 3246 glyphs it takes a while but completes.
Two issues though.
The reported amount of characters in the .xfont file is always 0x00010000.
And the compression ratio when I zip the resulting NotoSans-Regular_12_ASTC.glyph is really high:
1473 kB -> 96 kB

Sure, this is not fair since zip can work with the complete set at once.
But when I open the NotoSans-Regular_12_ASTC.glyph in a hex-editor it looks like this:

0000 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
0010 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
0020 66 68 18 80 46 18 18 10 18 10 7F 7F 28 10 04 28 fh..F.......(..(
0030 07 0A 11 00 00 80 5F 5C 61 33 13 20 84 00 11 42 ......_\a3. ...B
0040 66 00 00 B6 01 00 00 40 00 00 00 00 00 00 00 00 f......@........
0050 66 00 00 30 00 00 00 00 00 00 00 C0 00 00 00 80 f..0............
0060 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
0070 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
0080 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
0090 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
00A0 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
00B0 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
00C0 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................
00D0 FC FD FF FF FF FF FF FF 00 00 00 00 00 00 FF FF ................

Is this really compressed?


The example works, sort of, it displays the arrow.
But it deletes the Symbola_85_ASTC.glyph and Symbola_85_ASTC.xfont files in the resources dir when you start it.
Under "Content" it shows errors.




Title: Re: UTF-8 Encoding Error
Post by: Simon on April 30, 2020, 12:31:09 PM
The example also shows the arrow in my case. As flash content the fonts are displayed as .raw files. A self-generated .raw file brings the same result with the arrow. 
However, it does not work if I use my self-created .glyph or .xfont file, which is used to save the font data to the flash memory of the display.

Title: Re: UTF-8 Encoding Error
Post by: BRT Community on May 04, 2020, 04:41:08 PM
Hello,

Can you check the settings of EAB font conversion?   
To use the glyph in RAM_G,  you need set it properly in EAB.  See attached image.

Best Regards,
BRT Community
Title: Re: UTF-8 Encoding Error
Post by: Simon on May 05, 2020, 07:56:17 AM
Hello,

thank you for your answer. I don't want to use the glyph file in RAM_G but save it on an external flash and only load the xfont file into RAM_G. With this file I can then address the graphic data. I convert the data for the flash with the EAB Flash Utilities and load it via the map file into the ESE, just like I would do it on the hardware or have done so far.
This worked without any problems for me so far, only with the mentioned symbols it still does not work.
I have attached my setting of the EAB and the use of the font in the ESE. To show that my approach works basically I added some symbols from lower code pages.
Because the used symbols maybe hard to see in the screenshot, here is the code in text form:
Code: [Select]
CLEAR(1, 1, 1)
CMD_SETFONT2(0, 0, 0)
CMD_TEXT(59, 41, 0, OPT_FORMAT, "%c%c BC", 129091, 65)
CMD_BUTTON(51, 104, 100, 30, 0, OPT_FORMAT, "%c", 129091)
CMD_TEXT(25, 160, 0, 0, "♪☣⛇∰⌚🡃")

Regards
Simon
Title: Re: UTF-8 Encoding Error
Post by: pauljiao on May 08, 2020, 04:48:19 AM
Simon, as far as I knew, rendering the glyph from flash needs
1) The glyph is based on ASTC.
2) The starting address in flash shall be 64 bytes aligned.  From the screenshot, you may see the address 10000 is not 64 bytes aligned.

Title: Re: UTF-8 Encoding Error
Post by: Rudolph on May 08, 2020, 09:40:32 AM
When putting both the .glyph and the .xfont into the FLASH, the address setup in EAB is not used anyways.
EAB adapts the address contained in the .xfont file automatically to whatever the .glyph ends up at.

See: http://www.brtcommunity.com/index.php?topic=26.msg93#msg93

Okay, you still need to either load the .xfont file from FLASH to RAM-G or use the changed .xfont file that is written
when creating the .bin for the FLASH.
Title: Re: UTF-8 Encoding Error
Post by: Simon on May 08, 2020, 11:47:52 AM
Hello pauljiao,

thank you for your comments. I adjusted the addressing and repeated the test, but got the same screen output. If that was really the problem, shouldn't the output of the other icons have been broken as well?
Were you able to generate a working example?