BRT Community

General Category => Discussion - EVE => Topic started by: gt5659a on November 16, 2020, 05:13:42 PM

Title: Streaming AVI to VM816CU50A
Post by: gt5659a on November 16, 2020, 05:13:42 PM
My goal is to stream an AVI file to the BT816 to drive an LCD screen.  The AVI file will be about 40 MB, it's binary (black or white pixels) and it will change every 30 seconds or so, and each time I need to stream it to the BT816 with as little latency as possible (I want the video to start within a 0 to 2 second window of nominal start time).

I have a VM816CU50A eval unit (it has an LCD).  I am able to create an external AVI file, and stream it to the LCD screen with the EVE Screen Editor v3.3.0.  The timing appears to be acceptable--the latency is very small: it takes less than a second to start the display after I click the "Upload RAM and Coprocessor" button with the following code:

CLEAR(1, 1, 1)
CMD_PLAYVIDEO(0, "C:/Users/Public/Documents/EVE Screen Editor/Examples/testvideo3.avi")

So far, all is well with the eval board, it appears the BT816 can dow what I want and now I need to get the BT816 integrated into my global architecture (which is coded in C#).  Two ways that I know of to use the C++ code in C# is either a CLI approach or a DLL approach.  In either case, I think the first step is getting it to work in C++, so that's what I am trying to do now.

I tried the easy route to export from ESE (running as an admin): Export-->EVE Hal 2.0 Projects-->VM81650A_LIBFT4222.  This gives me an error stating "Error copying assets to project folder: [Errno 2] No such file or directory: untitled_BT81X_EVE_HAL/untitled/Test/testvideo3.avi")".  I can export projects from ESE and run them in Visual Studio without issues if there isn't a CMD_PLAYVIDEO in them.

I also hacked some code directly in C++ and was able to display one of the provided videos of Star Trek with Jeri Ryan (FYI my 13 year old self thanks you for this), and here's a snipped of such a project:

void Skeleton(Gpu_Hal_Context_t *phost)
{
   Gpu_CoCmd_FlashFast(phost, 0);
   Gpu_CoCmd_Dlstart(phost);
   App_WrCoCmd_Buffer(phost, CLEAR(1, 1, 1));
    #define ADDR_VIDEO 4096
    Gpu_CoCmd_FlashSource(phost, ADDR_VIDEO);
    App_WrCoCmd_Buffer(phost, CMD_PLAYVIDEO);
    App_WrCoCmd_Buffer(phost, OPT_FLASH | OPT_SOUND | OPT_NOTEAR | OPT_OVERLAY);
    App_Flush_Co_Buffer(phost); //Video plays after this
}


Here's my specific questions.

1. Is it worth digging into the Export from ESE approach (I opened up the python export file but gave up)?  I had hoped this would make it an easy way for me to get some sample code to get moving, but if it's going to be a lot of work I can just start to hack the C++ directly
2. Can someone explain the process that ESE uses to generate the video from the provided AVI file?  Specifically, does ESE break the AVI in to bytes and add them in after a CMD_PLAYVIDEO, does it alter the AVI format and send it to flash, provide a direct filename/location link to BT81x, etc? 
3.  It seems the C++ code that I have is very close to what I need but I don't understand how the avi file gets assigned to the "ADDR_VIDEO 4096"...I think this is a location in the flash memory but I don't think that the flash upload is fast enough to be able to display the video with the low latency I am seeing in ESE (e.g., the video starts in about 500 milliseconds regardless of the size of the video tested (up to ~80 MB) hence I don't think it's going to flash).
4. Can I assign the ADDR_VIDEO (or some equivalent) to be a file name like "XYZ.avi"? and then just reassign to the new AVI whenever I need to play the video?
5. Any other pointers on how to get this to work?


Any help would be greatly appreciated!






Title: Re: Streaming AVI to VM816CU50A
Post by: BRT Community on November 20, 2020, 06:04:39 PM
Hello,

Thank you for your questions.
I will look into why the export feature is causing errors when a CMD_PLAYVIDEO command is included.

For your first question, the export code feature can be very useful, however it would help to know what your target MCU is to be able to advise further.

Eve Screen Designer uses our Video converter utility (now incorporated into EVE Asset Builder (https://brtchip.com/eve-toolchains/#EVEAssetBuilder)) to convert the .AVI file into an EVE compatible format. A general use case is to convert the video and store this converted asset in flash/RAM_G, you can then use the CMD_PLAYVIDEO to play the video using either the OPT_MEDIAFIFO or OPT_FLASH option to tell the co-processor where the video data is stored and where to stream it in from.

I will need to contact the development team to query how Eve Screen Designer simulates using a video from flash memory as i do not believe that it currently supports uploading flash images to the development board.

When using flash as the source for your video file you require  '#define ADDR_VIDEO 4096'. This does denote the location in flash where the video file is stored and is used in the subsequent command:
   
      Gpu_CoCmd_FlashSource(phost, ADDR_VIDEO);

Unfortunately when streaming the video from flash you cannot set ADDR_VIDEO to be a filename.

I have attached a set of examples which covers various ways of streaming video with EVE.

Best Regards,
BRT Community
Title: Re: Streaming AVI to VM816CU50A
Post by: gt5659a on November 30, 2020, 11:59:32 PM
Thanks for the response, that helped clear things up quite a bit. I am not using an MCU, rather I am using Windows 10 with an FT4222 stream to the BT81x directly. 

I have tested out some of the sample code and was able to get bitmaps and JPEG/PNG to display correctly, but the frame rate was too low for that I need (I was able to stream about 5 frames per second for 800X480).

I am able to get 15+ FPS by manually generating an AVI file, deconstructing the AVI, and writing to memory. 

However, the AVI streaming via EVE Screen Designer (with CMD_PLAYVIDEO) is working faster (almost twice as fast)...any word back on the development team?  I would like to just steal that code since it seems to be working so much better than my current code :)

Thanks.
Title: Re: Streaming AVI to VM816CU50A
Post by: BRT Community on December 01, 2020, 11:39:49 AM
Hello,

Unfortunately I have not heard back from the development team yet, I will chase them up for you.
However, you can double check this by looking at the source produced by ESE, if it does program the onboard flash there will be a .bin flash file output and a FLASH_PROGRAM command in the source.

In general the EVE series of graphics controllers are not designed for video playback, however this functionality is supported. Depending on resolution of the video performance may vary. Are you seeing this 15fps when using RAM_G to stream the video data? In general storing the video data in flash vs RAM_G and using the OPT_FLASH opting with CMD_PLAYVIDEO will give the best results in terms of achievable frame rate.

Best Regards,
BRT Community
Title: Re: Streaming AVI to VM816CU50A
Post by: gt5659a on December 02, 2020, 03:32:20 PM
I am not loading anything to flash from ESE rather I am using the command buffer (CMD_PLAYVIDEO command and REG_CMDB_WRITE operations).  After some more code hacking I am able to get the expected frame rates for the very simple, binary image AVI files.

Some notes for others who may be tinkering with this:

1. You need to crank up the internal clock on the FT4222 (ft42Status = FT4222_SetClock(dmdControllerSpiBusHandle, FT4222_ClockRate.SYS_CLK_60)) to make sure that that front-end/input isn't the constraint
2. Similarly, this is running at 30MHz SPI clock (FT4222_SPIMaster_Init(dmdControllerSpiBusHandle, FT4222_SPIMode.SPI_IO_SINGLE, FT4222_SPIClock.CLK_DIV_2, FT4222_SPICPOL.CLK_IDLE_LOW, FT4222_SPICPHA.CLK_LEADING, 1)
3. Hopefully this doesn't enrage the Bridgetek gods, but I went off-menu/datasheet and used a pixel clock of 60 MHz (Gpu_Hal_Wr8(host, REG_PCLK,DispPCLK) with DispPCLK = 1) and verified this on a scope that you're getting ~60 MHz (this 60MHz seems the same regardless of the system clock setting of 60MHz or 72MHz)...testing a reworked eval board with a shiny new BT817 today to see if we can go faster  :)
4. The frame rate is cut in half when running from host PC, through an external, very high-end USB2 hub (UHR307) then to the VM816 eval board as opposed to connecting the eval board to the host directly (the hub is otherwise working fine so it may or may not be an issue specific to this hub)
5. The math for the VM816 eval board+LCD I have is (I think): 525 rows * 928 columns / 60 MHz PCLK = 0.00812 seconds per frame → ~123 frames per second theoretical.  Empirically, I am getting display times of about 11.1 seconds for an AVI file of 1500 frames (800X480 resolution, AVI file size ~30MB) → ~128 frames per second, but I can't visually see if every frame is actually being displayed but the math looks reasonable...and similarly when I scale it up to a resolution of 1080p and see frame rates of ~27 FPS (looks wonky on the 800X480 display however!)
6. Using this same code, "chickens-4.avi" plays in about 13.5 seconds as does the Star Trek video converted/streamed as AVI...my colleagues must be wondering why I get paid when they see Jeri Ryan and those chickens clucking around at my desk (and many other reasons)
7. Windows 10 OS, code in C++ and then ported over to C# (both in Visual Studio 2019)
8. C# has the same performance in terms of frame rate, and I'm using the FTD2XX_NET wrapper to interact with the FT4222 along with converted functions from the C++ example code (such as Gpu_Hal_Powercycle, etc.) which I just made C# equivalents of

Understood that this IC isn't designed for video streaming--this application would be replacing an FPGA so it's a pretty big win for cost and (more importantly) system complexity reduction.  however, do you see any reliability issues using a BT81x in video streaming mode of this nature (e.g., not using the flash which could, depending upon the specific flash IC, have some issues with the number rewrites it can support maybe?).  I will be streaming AVIs via the command buffer, and the AVIs are different every time so I can't upload to flash and hit the cycle times that are required.  If you don't think there would be any reliability issues in this mode I will proceed to the layout!

As a hopefully constructive criticism: While the support for implementation has been great thus far, it would be nice to get a bit more in terms of logical/physical design support.  For example, the Altium library fo the BT81x doesn't appear to be up to date, and I couldn't find any bill of materials for the VM816 eval boards nor their CAD design files etc. 
Title: Re: Streaming AVI to VM816CU50A
Post by: BRT Community on December 03, 2020, 01:37:16 PM
Hello,

Thank you for your comments and tips, i'm sure they will be useful to any other customers who happen upon this form post.

I will note that USB performance can be highly variable and is effected by the number and type of USB Peripherals enumerated on the system. The FT4222 also uses USB BULK transfers which are the lowest priority of USB transfers. The USB host controller is responsible for scheduling USB transfers and scheduling algorithms can vary across systems. This may account for some of the performance gains you see when not using the USB hub to connect the FT4222 to the VM816 module.

As for reliability issues, no I do not for see any issues when using the command buffer to stream AVIs to EVE. This was the method used for previous EVE generations (FT81X) and many customers have had good performance using this method.

I will raise your point regarding the Altium libraries to the hardware team. In general we don't release the BOM or layouts for our development modules. But if you have any specific questions concerning components used I can look these up for you.

Best Regards,
BRT Community