Building and live debugging the VLC Android app on a Windows 10 machine

Recently I've been looking for an Android media app so I can cast very large .mkv video files from network storage (Synology NAS) to a Chromecast.   I have tried various android apps but most can't handle huge video files residing on a network share.

Having loved VLC media player since back in Windows XP days I was excited to see they have an android app. I figured I'd give it a shot. It's had Chromecast support for a while and supports almost all video formats.

I had a few issues getting VLC to work at all with my Synology, since a while back I had disabled SMB1 after WannaCry and this led to an issue where VLC would just keep prompting for credentials over and over.  That was the initial motivation to try to debug the app. I have since learned that SMB2 support was recently added and I didn't have the update yet.

I've hit a couple other minor issues with the casting usability like if I pause the stream while casting, thirty seconds or so later the app will kill the stream and stop playback on the remote device.

Luckily VLC is entirely open source, so in theory I could debug it myself. (If I knew what I was doing ;)

The docs state that the build is only really supported on Linux.  That was a bit of an obstacle for me since I wanted to live debug the app over USB and my only suitable machine for this is my Windows 10 laptop with secure boot and BitLocker enabled. Dual booting Linux was not an option. Hyper-V virtual machines wouldn't help either as USB support is absent.

The repository has a lot of unix-y stuff, symlinks, heavy use of autotools, cross compilation using clang, etc.

My solution was to use Windows Subsystem for Linux for the VLC compilation but Windows Android Studio for any source editing and live debugging over USB.

Surprisingly, this setup worked! (after a lot of googling and trial and error)

Prerequisites

• A recent Windows 10 build, with (WSL) enabled, and Ubuntu 18.04 installed from the Windows Store.
• Awareness of this issue which also applies to regular Ubuntu 18.04. See my handy post here.
• Both the Linux and Windows versions of both the NDK and SDK. (I put them all under %localappdata%\Android, as sdk, sdk-linux, and ndk, ndk-linux.) I missed this at first and only had the Linux NDK and could not get the live debugging to work. I obtained the Linux version of the SDK from a Linux Android Studio install. (There may be easier way to get it).
• A lot of patience. Unfortunately building on WSL doesn't always work reliably. For me it would sometimes fail complaining about missing files or fail trying to mv a folder. In these cases simply rerunning the compile.sh script will work. It seemed to do better when running on a fast (SSD) disk. If I were to issue rank speculation I'd say it's probably due to either a WSL bug or design issue and these problems are related to how file access is performed in WSL vs a real Linux kernel.  It would be crazy to try to do this for a "production" build or anything other than experimenting and debugging.

Optional

Since I was starting from scratch a lot and making changes I created forks of the git repositories vlc-android, vlc, and medialibrary on a personal Gitlab server and updated the build scripts to use my server.

What follows is basically an adapted version of the linux instructions for building vlc-android that worked for me recently with this Windows 10 WSL Ubuntu setup...

Prepare (WSL) Ubuntu environment

Open an Ubuntu window and setup environment variables and path to get started:

#
# CHANGE TO SUITABLE PATHS FOR YOUR ENVIRONMENT!!
#
export ANDROID_SDK=/mnt/c/Users/steve/AppData/Local/Android/sdk-linux
export ANDROID_NDK=/mnt/c/Users/steve/AppData/Local/Android/ndk-linux
export PATH=$PATH:$ANDROID_SDK/platform-tools:$ANDROID_SDK/tools

The paths above point to the linux versions of the SDK and NDK that are mentioned in the prerequisites.

Install packages

sudo dpkg --add-architecture i386
sudo apt update
sudo apt install -y zlib1g:i386 libstdc++6:i386 libc6:i386
sudo apt install -y gettext automake ant autopoint cmake \
    build-essential libtool patch pkg-config protobuf-compiler \
    ragel subversion unzip git openjdk-8-jre openjdk-8-jdk flex python wget
 

My addition here is the gettext package which is missing from the default Ubuntu install and leads to a build failure if not present.

Set gradle.properties file to point to java 8:

echo "org.gradle.java.home=/usr/lib/jvm/java-8-openjdk-amd64" >> ~/.gradle/gradle.properties

This has been necessary on every machine I've tried. If not done, you'll get this error during the Java compilation:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':medialibrary:compileDevJavaWithJavac'.
> javax/xml/bind/JAXBException

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Setting gradle.properties here only affects the virtual Ubuntu environment since ~/ or home/steve is not visible to the Windows file system.  

Grab the source and try initial build

In the Ubuntu window, change directory to folder that will hold the repository, one that you can also open with windows, i.e. /mnt/c/Users/steve/AndroidStudioProjects

git clone https://code.videolan.org/videolan/vlc-android.git
cd vlc-android
./compile.sh -a arm64

The device I'm testing is a Pixel 3 so I'm using the -a arm64 switch. Use the appropriate platform for your device.  See compile.sh --help.  Since we didn't specify release or releaseSigned, debug binaries will be created.

Note

The compile.sh script will update the local.properties file at the base of the repository with the paths for ndk.dir and sdk.dir as set in the $ANDROID_xDK environment variables. Later when opening the repository with Android Studio for Windows  it will complain that the paths are invalid and reset them. (This can become a bit of a pain so a permanent solution is needed).  I was toggling them manually. But I'm a bit ahead of myself.

The first time the build will fail with the issue pointed to in the prerequisites section. After it fails, run this to add automake 1.15.1 support:

# still in vlc-android directory (base repository folder)
sed -i 's/check automake 1.15/check automake 1.15.1/' vlc/extras/tools/bootstrap
sed -i 's/AUTOMAKE_VERSION=1.15/AUTOMAKE_VERSION=1.15.1/' vlc/extras/tools/packages.mak
sed -i '$a\f0d4717ebe2c76cec5d487de090f6e1c0f784b0d382fd964ffa846287e2a364a52531a26ab98b7033ac04ed302a247b3b114299def54819a03439bfc962ff61b  automake-1.15.1.tar.gz' vlc/extras/tools/SHA512SUMS

Run the build (for real this time)

Now that that's out of the way, it's hopefully smooth sailing from here on out:

./compile.sh -a arm64

If you luck out, an hour or so later, you might just have a completed and packaged build.

Using Windows Android Studio

Now that the build has completed we can launch the debug version on the Android device and step through it and potentially make changes.

Open the vlc-android project in Android Studio.  You'll get the warning (eventually) that the local.properties file doesn't have a valid path and it will be corrected.  Open the file and check that both ndk.dir and sdk.dir are correct and pointed at the Windows versions this time!

Check the Build Variants UI and select dev config where possible so it matches the compile.sh build we ran earlier:

Do a gradle sync and from here you should be able to live debug the java or kotlin code without any further config, but if you want to step through any native c or c++ code there are a few more steps.

We need to tell Android Studio where the "un-stripped" debug binaries reside. Apparently even in debug/dev mode the packaging process strips the symbols (by default.)  

Even when the symbols get loaded properly, we still have an issue where the source code file paths will be wrong.  Remember the source paths will be in Linux style relative to the WSL filesystem like /mnt/c/Users/steve/AndroidStudioProjects which Windows won't know about.  Luckily there is a setting to map the paths between the ones that existed when the binaries were built to the ones we have now.

Edit the debug configurations:

Under Debugger  set Debug type to Dual (both java and native debuggers are launched), then under LLDB Startup Commands:

Enter these settings (one per line, adjusted for your environment):

settings set target.exec-search-paths C:\Users\steve\AndroidStudioProjects\vlc-android\.dbg\arm64-v8a
settings set target.source-map /mnt/c/Users/steve/AndroidStudioProjects/vlc-android C:\Users\steve\AndroidStudioProjects\vlc-android

The first setting tells lldb where to look for debug binaries. The compile.sh script has dumped them to the .dbg/ folder in a subfolder named for the ABI.

The second setting does the source path mappings between the build paths vs Windows paths.

You can verify the debuggable binaries have loaded by breaking into the native debugger, going to the native debugger/ lldb tab:

and entering the lldb command image list. You should see output like this:

(lldb) image list
...
[212] 6BA64A04-6CBC-9178-4574-C2ACEA1E6A00-57B7AEFE                    C:\Users\steve\AndroidStudioProjects\vlc-android\.dbg\arm64-v8a\libvlc.so 
[213] EB342DD4-52E0-74F7-8F8B-7EAD23E17464-EF25A5B6                    C:\Users\steve\AndroidStudioProjects\vlc-android\.dbg\arm64-v8a\libvlcjni.so 
[214] 7EF2A101-8690-D83D-783E-B24174D09EEE                    C:\Users\steve\.lldb\module_cache\remote-android\.cache\7EF2A101-8690-D83D-783E-B24174D09EEE\gralloc.sdm845.so 
[215] BFAC81E3-05D6-51BE-817B-ABADA072A117-F91941C8                    C:\Users\steve\AndroidStudioProjects\vlc-android\.dbg\arm64-v8a\libmla.so 

Notice our project's libvlc, libvlcjni, libmla libraries are loaded from the .dbg folder.  Most other modules are loaded from .lldb\module_cache\remote-android.  If you get this far you should be ready to set and hit breakpoints in native code: