Blogging the Monkey

ubuntu-tv 1080p on omap4 panda

2012-01-14T19:13:00.001-08:00

you can find a higher resolution video at: http://www.youtube.com/watch?v=HJuyNrVOS1I
needs a bit of cleanup, but patches are here: https://github.com/robclark/qtmobility-1.1.0

update: and as usual Ricardo has a nicer looking video at his blog

omap5 at CES

2012-01-12T20:27:00.000-08:00

http://www.engadget.com/2012/01/12/ti-omap-5-exclusive-demo-laptops-ultrabooks-ces-2012-video/

xbmc update

2012-01-05T20:29:00.000-08:00

of course I've been too busy to write anything, but Ricardo has updated his blog w/ a note about xbmc progress:

http://rsalveti.wordpress.com/2012/01/06/hw-video-decode-and-xbmc-ubuntu-linaro/

Catchin up..

2011-11-27T11:32:00.000-08:00

just catching up on some news since last posting:

First, TI/OMAP PPA for ubuntu 11.10 now contains support for hw video codecs via DCE and gst-ducati. (decoders: h264, mpeg4, mpeg2, vc1; encoders: h264, mpeg4). Yah!

But lately I've been mostly working on omapdrm, a DRM/KMS display driver for omap, corresponding X11 driver (xf86-video-omap). The kernel driver is now queued up in the staging tree for 3.3. But not forgetting multimedia, I've been also working (as a linaro assignee) on extending dri2 protocol for more efficient video rendering (see linux-video.pdf) and UMM/dmabuf for sharing buffers between multiple devices (camera+drm, or multiple drm devices for a prime type setup).

And lastly, been doing some hacking trying to get xbmc working nicely with the hw video codecs for hw accel hd playback.. but more on that shortly when I have something work.

Building DCE firmware

2011-06-23T13:36:00.000-07:00

Now, thanks to public release of codec libraries, and after some slacking on my part, all the bits and pieces needed to build your very own ducati (cortex-m3) firmware are available. I've put together a wiki page with instructions of where to find all the pieces, and how to build, for 2.6.38 kernel:

http://www.omappedia.org/wiki/DistributedCodecEngine

This is using syslink-2.0, tiler-2.0, and GA codecs/FC/etc.. but now at least, if you want to use a kernel with a different version of syslink, you can rebuild the firmware with appropriate corresponding bios-syslink on the coprocessor side.

better late than never

2011-04-17T12:09:00.000-07:00

haven't had time to post for a while, so just getting caught up on a few things (in reverse chronological order)

ffmpeg vp8 decoder

Mans Rullgard has improved the original neon vp8 patches, and pushed them into the main tree:

VP8: ARM NEON optimisations for dsp functions

omap drm/kms display driver

A while back I started experimenting with the DRM display driver framework, and now have a basic driver which implements the KMS part of DRM. It uses a plugin API for SGX/PVR driver to register and handle it's own set of ioctls related to 2d/3d acceleration. Still TBD is overlay support, and cleaner way to handle buffer allocation (GEM?).

Now with the userspace pvr xorg driver, basic XRandR is working (change resolution, setup multi-monitor virtual display, etc). Being able to change resolution without cryptic sysfs cmds is nice for a change.

universal buffer allocation/management BoF

There was a BoF at ELC last week on the topic of common buffer allocation/management APIs to support zero copy buffer passing between various IP blocks (display, GPU, codecs, ISP, etc). Currently each SoC vendor has some custom API (CMEM, PMEM, NVMEM, TILER.. etc). Google is introducing ION. Most of the rest of the linux world (ie. desktop) uses GEM and/or TTM, which admittedly are somewhat GPU-centric.

In the desktop world, 3d/codec accelerators and display are all on the graphics card. But in the embedded/SoC world, you might have several vendors who use a common 3d block (for example), but each with their own unique display controller. And different video encode/decode accelerators. And different ISPs.. and so on.

For me, right now GEM is interesting as a way to expose allocating of TILER buffers on OMAP4 for video encode/decode and display. DRI already provides a path in userspace to pass GEM buffers and use DRM to handle the authentication duties (although GEM/DRM are perhaps not strictly required.. but they are something that exists in upstream kernel tree today). But short of mapping buffers into userspace process, there is currently no good way to pass these buffers to a v4l2 camera, or IVAHD video encoder/decoder. Possibly interface to video encoder/decoder IP can be thru the DRM display driver (another plugin, perhaps). Although that still leaves camera.

And there was also a bit of discussion on the related topic of how to expose display to userspace.. fbdev is ancient legacy, v4l2 MCF is the new kid on the block, but DRM/KMS is what is used in the desktop world. It seems like MCF should be more flexible for building different sorts of graphs, and to handle oddball features like writeback-pipe on OMAP4. Although DRM/KMS is already handling hotplug, EDID parsing, and provides sufficient flexibility for building display graphs (fb -> crtc -> encoder -> connector). At this point I prefer sticking with DRM/KMS for mode setting so that normal uses can be exposed to userspace in normal ways.

At this point, it isn't clear what the conclusion will be. A more modularized DRM with buffer management more easily split out (or at least shared with other devices)? ION or GEM or some merger of the two? The BoF was just a short 1hr session to better define the problem. The next step will be follow up sessions during the Linaro Developer Summit in Budapest.

Announcing libdce and gst-ducati

2010-11-27T09:15:00.000-08:00

And now it is time to take the wraps off of a weekend/evening project I've been working on for the last couple weeks. Last weekend, I wrote libdce which is a small library to allow for remotely accessing the codec-engine API from linux (without needing OpenMAX). And in the last few evenings plus weekend, I wrote a GStreamer plugin, gst-ducati, to decode video using the codec-engine via libdce.

So far, the following decoders are working:

ducatih264dec - H.264 (bp/mp/hp)
ducatimpeg4dec - MPEG-4
ducativc1dec - VC-1 (sp/mp/ap)
ducativp6dec - On2 VP6
ducativp7dec - On2 VP7

(The H.264 and MPEG-4 decoders are provided in libdce/firmware/dce_app_m3.xem3 which replaces base_image_app_m3.xem3.. for the other codecs, contact your TI representative.)

Why do this, when (a heavily patched) gst-openmax is providing support for hw accelerated decoding? The main answer is that I thought it would be a fun project, and it gives me a way to play with the codecs directly without OpenMAX getting in the way.

Currently libdce is working with L24.9 versions of syslink and kernel, since that seems to be the versions commonly packaged. But next step I'll update libdce to L24.11 for those who like the bleeding edge.

---

Update: latest master libdce is working on L24.11.. for L24.9 use the corresponding git tag.

enna + gst + omap4

2010-09-20T18:03:00.000-07:00

So.. I've been thinking for a while now that a great project would be an omap4 based media-center-pc. Hardware accelerated 1080p playback (including h264 high profile), dual cortex-a9 SMP, fast 3d acceleration, etc.. quite a worthy replacement for my aging, (ahem, umm.. "upgraded") appletv.

With that in mind, I found enna from the geexbox project. (Check out the slick looking screenshots.) It is built on the collection of EFL libraries for slick looking user interface (with both unaccelerated backend and gles backend for faster UI). It uses libplayer as the backend for media player(s), and libvalhalla for clever stuff like extracting and downloading media metadata (such as album/dvd cover art and info) automagically.

Since I'm currently working with an ubuntu filesystem, I decided to skip for now building the entire geexbox filesystem, and instead just build the equivalent versions of EFL libs, plus enna, libplayer, libvalhalla, etc, for ubuntu. The advantage being that I already have here setup gdb, oprofile, and all the other tools that I find so useful. Plus, it is the filesystem I'm working with day to day, so I don't have to constantly rebuild the userspace components of our multimedia stack for two different filesystems on a regular basis.

Enna's default theme, out of the box, has plenty of eye-candy and effects.. but is a bit sluggish unaccelerated. Especially because the elm_slideshow widget seems to be somehow causing the background images to be repeatedly rescaled, instead of cached.. it doesn't help that the smooth rescale code in evas is not neon optimized. But hopefully getting the SGX package installed and gl backend running will fix all that. But now, as of yesterday, there is a new 'stb' theme which is much more lightweight and fairly snappy without acceleration.

Since libplayer has a GStreamer backend, the goal would be to have accelerated video playback in enna. Although the current gst libplayer backend is experimental. But I've started hacking away to try and get the gst backend into good shape. The results of what I have are here:

http://github.com/robclark/libplayer/

video/audio playback is now integrated and working, and things like pause/seek seem to basically work. Some of the more advanced stuff like TV turner or DVD navigation are missing. But it's a start.

My here is my handy script to build this all: build-enna.sh

And here is my config file (to go in ~/.enna): enna.cfg

(hint: click on the raw or download links)

------

update: the libplayer patches for better GStreamer support have been merged a couple weeks ago.. so you can pull libplayer from geexbox/openbricks tree.

ffpv8 + neon = 720p24

2010-08-23T20:13:00.000-07:00

btw, been a long time since I had a chance to update the blog.. so I just thought I'd drop a quick note about something I've been playing with for the last few weekends.. the new ffvp8 decoder!

I've started writing the neon dsp functions for the VP8 decoder, as an excuse to learn a bit more about sw video codecs, neon, and VP8. At this point, not all of the dsp functions are implemented, but all the important ones for all the VP8 clips that I can find are implemented (loop filter, bicubic MC functions, and some misc other functions). Most of the major other ones, such as the bilinear MC functions, don't seem to be used in the clips that I can find, but should not be too hard to add when I find clips to test with.

The result is some 15-20% faster than libvpx, mostly thanks to ffvp8 being more cache friendly than libvpx decoder, and not doing silly things like memcpy of reference frames, rather than my hard-core neon optimizing skills.. and this is even without ffvp8 being a multi-threaded decoder, which is something that would benefit an SMP cortex-a9 platform like OMAP4 if done properly. And all this should be possible to get a bit faster by spending some time tweaking the instruction order to avoid stalls and some other tricks like that. (And hopefully I'll learn a few tricks in the process as the patches are reviewed.)

The result so far is here:

http://github.com/robclark/ffmpeg/commits/vp8-4

Current status is that it is all working, and producing bit exact output compared the plain 'C' versions of the DSP functions for all the test clips I have. I'll update again when I add more or when the patches are in upstream ffmpeg.

I also have some work-in-progress patches for gst-ffmpeg to avoid a memcpy for codecs that don't support edge emulation, although these depend on rowstride and some of the other related features that we've added to GStreamer for omap4.

git format-patch for specific commit-ids

2010-03-08T11:38:00.000-08:00

for revision ranges, appending ^! (which appropriate escaping) to a commit-id causes it to represent that range beginning/ending with that commit-id inclusively instead of exclusively. So for specifying a revision range consisting of exactly one specific revision (let's call it r1), you can specify r1^! r1

And if you want to generate a patch from all your commits while ignoring all other commits:

for c in `git log --author=Clark | grep ^commit | awk '{print $2}'`; do

git format-patch "$c^\!" "$c";

done

texture streaming in action

2010-02-17T06:51:00.000-08:00

Hands-on with TAT's dual-screen phone concept and augmented reality app

texture streaming

2009-10-08T18:13:00.000-07:00

IMG has an interesting texture streaming EGL extension for their GFX graphics core, which is supported on TI omap3 devices. It provides a way to stream YUV (or RGB) video to a 3d surface without incurring the expense of a texture upload or an extra memory copy and without the need for colorspace conversion. This makes for interesting possibility, such as decoding video to a 3d surface.

The texture streaming extension utilizes a kernel API to tell the GFX hw when a new frame of video is ready to render. The catalog group in TI had created a 'bc_cat' linux kernel module to allow userspace to allocate buffers to use in texture streaming.

I've taken this driver, and made a few tweaks, and created a GStreamer video sink plugin (see gst-plugin-bc which includes the 'bc_cat' kernel module) to allow for using texture streaming in a GStreamer pipeline. This allows our accelerated codecs to decode video directly to the buffer used by the GFX core. The hw/DSP and GFX core do all the work:

gst-launch filesrc location=/mnt/mmc/iron_man-tlr2_h640w.mp g ! avidemux name=d d.video_00 ! queue ! omx_mpeg4dec ! queue ! bcsink d.audio_00 ! queue ! omx_mp3dec ! alsasink

Here is another example, this time using camera to capture to 3d surface:

gst-launch v4l2src ! "video/x-raw-yuv,width=640,height=480" ! bcsink sync=false

(sorry about the low quality video and the glare which makes it a bit hard to see on the screen)

Touchbook!

2009-10-08T18:10:00.000-07:00

I got my touchbook last night! Haven't had too much time to play with it yet, but a very cool little device.

NEON is fashionable

2009-08-12T17:22:00.000-07:00

... as long as we aren't talking about your wardrobe..

so, my college Daniel brought to my attention a gstreamer use-case that was in need of some performance optimization. When decoding DVD content, the audio samples from the AC-3 decoder are in float-32bit format. These need to be converted to 16bit integer format to play through alsasink. But the overhead of audioconvert on the ARM was quite high. Which seemed like a good enough excuse to learn NEON. So last weekend I broke out oprofile and had a look at where the cycles went, and what needed to be optimized. The pipeline I used for testing was:

gst-launch filesrc use-mmap=true num-buffers=12000 location=/my-clip.vob ! dvddemux name=d d.current_audio ! a52dec ! audioconvert ! audio/x-raw-int,width=16,depth=16! fakesink

Initially, this pipeline took roughly 20.7sec. By comparision, the same pipeline without audioconvert:

gst-launch filesrc use-mmap=true num-buffers=12000 location=/my-clip.vob ! dvddemux name=d d.current_audio ! a52dec ! fakesink

took roughly 11.1s. A couple functions immediately stood out: gst_audio_quantize_quantize_signed_tpdf_none() and audio_convert_unpack_float_le(). The former was particularly bad just due to the random number generation for dithering, which contained a divide by 0xffffffff. As Barbie (and processors everywhere) say, division is hard. Many embedded processors will emulate division (read: expensive), and even processors that have divide instructions, the cycle count is high. Just changing this to 32bit right shift (ie. divide by (1LLU<<32)) made a big improvement. Results might not be exactly the same, but it is close enough that you couldn't hear the difference. In the end, I left gst_fast_random_int32_range() untouched, but replacement vectorized code kept the >>32 approach.

The next step was to come up with a way to plug in accelerated versions of various audioconvert functions. I changed them all to use __attribute__((weak)) which is a neat trick of the GNU compiler (and ARM ltd. compiler) which lets you provide default versions of some symbol which can be overridden at link time. (Unfortunately this doesn't seem to be supported by all compilers that gstreamer supports, so I need to come up with a more portable approach.) This let me add an armv7 specific file to re-implement these two functions. The algorithms themselves stay basically the same, but are processing four elements at a time, except for a few instructions which use 64bit math and those are processing two at a time. In the end, the pipeline with audioconvert dropped to ~12sec, roughly 10x improvement for audioconvert (through a combination of processing 4x samples at a time, faster psuedo-random number generation, and the fact that NEON provides a saturating addition instruction). That is with gcc NEON intrinsics. I guess a few cycles could be saved by writing it all in assembly, which was my original plan, but at this point these two functions only show up a couple pages down in oprofile output.. so time would be better spent looking into liba52 (audio decoder). The video part of a playback pipeline is no issue, the DSP on OMAP3 can decode this without breaking a sweat!

Here is the patch. In the end, probably the long-term solution for audioconvert would be to use orc, for cross-platform vector acceleration. Although currently orc doesn't support floating point, and doesn't have a free NEON back-end. I may clean up my patch (to address the portability issues, and make the build system figure out whether or not to build my new armv7.c file depending on the target architecture). In the end, it depends on how much of a short-term solution that would be..

update: I thought I'd add a few links that I found useful

some Cortex-A8 / NEON info from TI wiki
NEON and VFP Programming section in ARM compiler tools assembler guide, gives a good reference on all the NEON/VFP instructions
ARM NEON Intrinsics section from GCC manual (but not really any more info compared to what you can find by just looking at arm_neon.h).. but once you know the instruction you want (from previous link) this is useful to find the matching intrinsic name.
Using NEON Support appendix from ARM compiler reference guide.. the GCC intrinsics pretty much match ARM's, and the ARM doc has more useful info

Sending GIT patches via email from behind oppressive proxies

2009-08-10T07:24:00.000-07:00

I was wanting to setup a wait to email patches via my gmail account.. I found this post from my friend Nishanth: Nishanth' tech rambles: Setting up Email forwarding System for GIT, which was quite helpful.

On my macosx laptop, sendmail worked out of the box.. but it finds the SMTP server from DNS. So at work it was using an internal SMTP server. And at home, it wasn't finding any SMTP server. And for some reason using the internal SMTP server seemed to only work for sending patches to internal addresses. I'm sure the built-in SMTP client could be configured to work as I needed, but I really had no desire to figure out how.

Following the instructions on Nishanth's post, I setup msmtp. (go macports!) And it works great. Somehow it automagically works properly both behind the proxy at work, and from home, depending on the systemwide proxy settings. (The power of Steve Jobs compels it!)

To send patches, I use this alias:


alias gsend='git send-email --from "<addr>" --envelope-sender "<addr>" --smtp-server /opt/local/bin/msmtp'

(replace <addr> with your email address)

silly programming games

2009-08-09T17:26:00.000-07:00

catching up on some old news... since that's easier than making up new news..

A while back, my buddy Mike decided it would be fun to write a quine. It was clever. But one good quine deserves another (or something like that). But I couldn't think of any approach for a 'C' quine that hadn't already been done in some form or another. And the empty file <insert-your-favorite-script-language-here> quine was too easy. So I wrote a self-disassembler in ARM asm: quine-1.S. Technically, it doesn't do any file I/O (if you ignore the fact that the program loader mmap's executable files / shared libraries), so I don't think it is cheating. If you are curious, the original with comments (slightly easier to follow) is here: quine-0.S. If you happen to have some sort of ARM platform running linux, the comments in the second file should explain how to compile it. If you don't, go out and buy one of these.

First post..

2009-08-09T17:18:00.000-07:00

Well, after resisting for so long, I decided to join the 21st century and start a blog... lets see how long this lasts..