B4J Library jAudioStreamer library

Hello,

This is my tentative to build an AudioStreamer library for B4J, fully user configurable.
I'm not a Java developer, so please excuse me if something is not implemented using best practices.
Was build for my own use, but may be helpful for others too.

You can find attached a sample code in B4J that allows you to test different parameters.
A loopback is created emulating network latency and jitter (using localhost).
Was tested on Windows , Linux, Raspberry Pi OS and Chromebook (Linux subsystem). The screenshot below is from a Chromebook, using an USB headset.

The library is pretty straight forward, similar with the one for B4A, but highly customizable.

Classes

aDevice

• Fields
  • devID - audio device ID
  • lineID - line ID for the current device
  • devName - audio device name
  • devDescription - audio device description

AudioStreamer

• Functions
  • Initialize(EventName as String, recID as Int, playID as int, samplerate as int, bits as int, chans as int, recbufsize as int, playbufsize as int)
    • EventName - EventName
    • recID - recording device id
    • playID - playback device id
    • samplerate - in Hz
    • bits - number of bits per sample (8 or 16)
    • chans - number of channels, 1 for mono, 2 for stereo
    • recbufsize - Line In buffer size. If 0 is used, then an automatic value is used
    • playbufsize - Line Out buffer size. If 0 is used, then an automatic value is used (usually = samplerate)
    • return true if AudioStreamer was successfully initialized
  • clearQueue - clear playback queue
  • flushPlayer - flush player buffer
  • flushRecorder - flush recorder buffer
  • IsInitialized - check if AudioStreamer is initialized. Return true if initialized, false otherwise
  • isPlayerAvailable - return true if Player thread is active
  • isRecorderAvailable - return true if Recorder thread is active
  • muteLineOut - if true, Line Out is muted.
  • PlaybackDevices - return a list of aDevice audio output devices (see aDevice class)
  • RecordingDevices - return a list of aDevice audio input devices (see aDevice class)
  • startPlayer - start Player thread.
  • startRecorder - start Recorder thread
  • stopPlayer - stop Player thread
  • stopRecorder - stop Recorder thread
  • writeData(buffer() as byte) - put a byte array to the player queue
  • validateIP(IP as string) - return true if the string represent a valid IP v4 address.
  • AllLocalIPs - return a list containing all locally available IP v4 addresses.
• Public variables
  • LineInBufferSize - (read only), currently used Line In buffer size. Can be set through Initialize. Value can be different from the one in Initialize, because there is a maximum possible value. If you set it bigger than the maximum allowed, maximum allowed is used.
  • LineOutBufferSize - (read only), currently used Line Out buffer size. Can be set through Initialize. Value can be different from the one in Initialize, because there is a maximum possible value. If you set it bigger than the maximum allowed, maximum allowed is used.
  • RecorderBufferSize - (read/write), the size of the read buffer (how many bytes are read at once). This can be useful if you want to encode the audio stream and need a specific length.
  • VolumePlayer - (write only) set the attenuator for playback (0-100, 0 = -80dB, 100=0dB)
  • VolumeRecorder - (write only) set the attenuator for recorder (0-100, 0 = -80dB, 100=0dB), most of the audio input devices does not support this.
  • queueLength - (read only), return the playback queue length

I am open to any suggestions or observations.

Current version: 1.0
Attached files:
jAudioStreamerLibrary.zip - the library (jar and xml files)
jAudioStreamerExample.zip - a sample application (see picture above)

 

[max123]

Hi @yo3ggx,

I've done something similar but using UDP transport protocol.
It works between ESP8266-ESP32 microcontrollers (transmitter and receiver) and B4A or B4J (only receiver for now).
On receiver side I can set parameters. It worked well at various sample rates, 44100, 48000, even 96000 Hz 16 bit Stereo.
The app started with support of 192 khz sample rate and 16, 24, 32 bit depth, but no yet implement some of these.
The audio at 96 Khz 16 bit is good, no problem for missing packets, and it have a better range than A2DP bluetooth audio.
The latency seem not bad, but I've not implemented a feedback to know it in a precise way.

 

[max123]

@yo3ggx I want to implement in my UDP streamer a loopback to test the real latency device-device, not just using localhost.
On this direction I've some ideas but never doing this on audio stremer.

My idea is to send some packets and the receiver send it back to the transmitter, then divide (on transmitter side)
the elapsed time by 2, so time to send and time to return back, this should work like ultrasonic sensors know linear
distance from objects using the the ultrasonics and calculate the time that brust return back.

Can I use this or I'm wrong ?
I can ask how did you implented it ?

In my implementation know and have a low latency is essential, because I will use this audio streamer for video call
between ESP8266-ESP32 and B4A/B4J and here even video trasmission/reception is involved.

A note. My implementation is just in code now (no library) on both sides, microcontroller and desktop device,
but probably I will write on both sides a same library that help to manage it using just some commands to
manage the stream.

Thanks

 

[stevel05]

That will give you the average time for the two way communication which may be enough. If you're troubleshooting, you might want the return message to include the time the original message was received and the time the response was sent so that you can calculate the time for outbound and inbound messages.

 

[max123]

Thanks @stevel05 for advices.

I wrote something similar on one of my ESP8266-ESP32 libraries. The library send a brust of packets, then the receiver sent it back, essentially ping 100 times the remote device in just a portion of second, then calculate the exact ping time expressed in microseconds. It is very stable and realiable value. I ping it 100 times because I seen that first UDP packets have a bigger latency time expecially when UDP is in idle mode (eg. not send data continually).

On my network I've around 4 ms ping time, so just 2 ms until the message arrive to the receiver, other 2 ms to retun back to the transmitter.

I even wrote some code that draw continually (on a TFT screen) a graph of the ping time and even show it numerically with min, max and average. This approach is not bad.

Yes, as you says, probably there is a gap of time between the receiver receive data and when it resend back, but this should be a very small time, but to be more precise it should be subtracted to the elapsed time result on the transmitter side.

Yes is even a good idea put the timestamp on the message that return back, to know it better.
On my UDP audio streamer I want implement it in microseconds, microcontrollers have a very precise time stamp, even better than 1 microsecond, but on desktop side this is not possible, and not accurate, so probably I will just use milliseconds.

 

[yo3ggx]

My idea is to send some packets and the receiver send it back to the transmitter, then divide (on transmitter side)
the elapsed time by 2, so time to send and time to return back, this should work like ultrasonic sensors know linear
distance from objects using the the ultrasonics and calculate the time that brust return back.

As you use UDP, you must mark the packets somehow, as the order you receive them can be different than the one you send them, especially over less reliable network connections. More, some packets may be lost. Usually RTP is used over UDP. In my case the latency was not so important, so I choose to use TCP and combine audio and data over the same TCP connection.

 

[max123]

As you use UDP, you must mark the packets somehow, as the order you receive them can be different than the one you send them, especially over less reliable network connections. More, some packets may be lost. Usually RTP is used over UDP. In my case the latency was not so important, so I choose to use TCP and combine audio and data over the same TCP connection.

Yes, I already read this more times, but I had no problems, no with missed packets, no for packets order, at least my audio is not bad on receiver side, at least
on my local network.

I tried some things with it, just to test:

- tried to read .wav files from microsd on ESP32 and receive on Android or desktop device with my simple app worked well, audio is good, no audible dropouts, up 96 Khz 16bit Stereo.
- tried the same thing but receiving on another ESP32 with audio DAC PCM5102A attached and headphones, here same results, audio is not bad.
- tried to receive the stream with VLC (only on Linux, not tested on Windows), here I had some problems and a lots of dropouts, probably I have to set the same buffer size that the transmitter.
- tried on Linux to receive the stream with just netcat (in just one line on terminal) and redirect the stream to ALSA audio driver. Here I can open programs like qjackctrl and Pathage and route graphically my audio to other programs that can record, or even add realtime effects and more, the audio is good.
- tried to read .wav files from microsd on ESP32 and receive on VLC over intermet with my friend at 1500 km from my house. Here I cannot listen the audio, but my friend says that sometime is good, sometime have a lots of dropouts.... but with VLC do the same here in my local network, so next I will try to send my small receiver app (android or desktop) to my friend and when have some time we test it better.

It is important that the receiver buffer size mathes exactly the transmitter buffer size.

Here I'm starting from the assumption that if I don't hear any audible dropouts, theoretically there shouldn't be any lost or reversed packets, but to know it better I have to search a way to send a progressive packet number inside any packet and extract here on the receiver side to check if some packets are missed or arrive in different order.

 

[yo3ggx]

I suppose VLC is using RTP/RTSP for audio over UDP. It is not enough to just send the packets one by one. Is like you are trying to play a wave file without a header. It works in your setup as you know bitrate/bits per sample etc. at both ends, so you can use raw data.

 

[max123]

I suppose VLC is using RTP/RTSP for audio over UDP. It is not enough to just send the packets one by one. Is like you are trying to play a wave file without a header. It works in your setup as you know bitrate/bits per sample etc. at both ends, so you can use raw data.

Yes, probably you are right.
In fact only with VLC I had problems, with my app that read raw data and just send to the audio streamer it works well.
Even using netcat probably it just read raw data. Netcat just start UDP server to listen, then pass raw data to the ALSA driver audio buffer. Here no RTP/RTSP involved.

Here my comments on the ESP8266-ESP32 UDP Transmitter and Receiver sketchs. This is part of my ESPAudioClass audio library I developed for these microcontrollers using Arduino platform. This show commands I used for Netcat and VLC to receive the audio stream.
I managed it some ages ago, now that I read my comments I see that it even worked to stream 192 Khz 16 Bit Stereo audio as well, that are a lots of data.

/*
   SD Wave file stream over UDP. Transmitter. (Client)

   This sketch read Microsoft Wave files from microsd and stream over network
   using UDP connectionless protocol. It is even possible to send broadcast to
   all local network devices. Combine both mode it is possible to stream to another
   device over internet, then the receiver (Server) will stream data to broadcast
   devices on other local network.

   Needs an UDP listener, another ESP or Netcat, VLC etc... that listen on port 8266 or 3232

   Listener can be another ESP programmed to listen UDP packet stream and send it to audio out
   or Android or Desktop app, or just a network media streamer like Netcat or VLC, if this is
   a case on Linux start the UDP listener with the follow terminal commands:

   Netcat:  nc -u -p 8266 -l | play -t raw -r 44100 -b 16 -c 2 -e signed-integer -
   VLC:     vlc --demux=rawaud --rawaud-channels=2 --rawaud-samplerate=44100 udp://@:8266

   NOTES:

   - Using Netcat your audio pass through the linux ALSA driver, it can be routed to any audio
     application and even processed with multiple effects at same time or can be sampled from
     audio programs like Audacity, Ardour5 etc...

     To do audio routing on linux we use Jack Audio Connection Kit, precisely we use QjackCtl or Pathage
     (that offers a frontend GUI for Jack and ALSA audio drivers) to route ALSA audio to Jack audio and
     optionally process it with realtime effects or just analyze with audio analyzers.

   - If you use another ESP as listener you can adjust the volume by connecting a potentiometer on any ADC
   - If you use VLC as listener, to adjust volume, use VLC volume control on the right bottom interface,
     or just use a linux master volume control
   - If you use Netcat, use a linux master volume control, you can even use alsamixer

   Tested successfully on Wemos D1 Mini @ 44100Hz, 48000Hz, 96000Hz, 192000Hz 16Bit Stereo samples

   This code is released to the Public Domain by Massimo Meli
   Please read Licence.txt provided with ESPAudioClass library
*/

/*
   Stream received UDP audio data to external I2S DAC.

   Demonstrates the use of the ESPAudioClass library for ESP8266 and ESP32

   Hardware required :
   - Any ESP8266 with I2S pins exposed, eg. NodeMCU, Wemos D1, Wemos D1 Mini, Wemos D1 Mini Pro, or any clone,
     on ESP-01 you can only use Sigma-Delta output on RX pin because it do not have others I2S pin exposed.
     Any similar board with ESP32 Extensa Chip.
   - One or more 44100 Hz, 16 bits, Stereo wave files in the root directory of the SD card, use short names 8.3 dos format.
   - An audio amplifier or just headphones to connect to the DAC output (we tested on PCM5102A, any DAC can work)
   - One or two speakers to connect to the audio amplifier if you do not use headphones.
   - One potentiometer if you would to manually set the volume.

   IMPORTANT NOTES:

   - Set the SD SPI around the max speed possible

   - Compile @ (160Mhz on ESP8266) or (240Mhz on ESP32) because can be CPU intensive so need to work as fast possible

   - if you do not have DAC you can get output from RX pin, the library use even Sigma-Delta output, or if you use ESP32
     can output from internal DAC0 & DAC1 (GPIO25, GPIO26), but this way you reduce the audio quality because dacs are not
     16 bits. If use Sigma-Delta output you just need to add a passive filter to the output pin. Note that this
     way you have MONO sound output only, for best quality we suggest to use a small DAC board.

   - Start with low volume, if you use an amplifier, lower the volume before ESP start executing the sketch,
     if you use headphones you may do not have a possibility to set the volume, set it with the library in the
     Audio.prepare() method, you can set the volume from 0 to 1024, may put it on the midle 512 for first tests
     then you can increase it. Values > 1024 do not overflows, just the library limit the output so you can't have
     distort sound. The best and faster way to control the volume is to connect a potentiometer to analog input (A0)
     and pass the ADC value when use Audio.prepare method, you can even use Audio.getVolumeL(), Audio.getVolumeR()
     to get it.

   - This sketch was tested with 44100 Hz, 16 bits, Stereo wave files on Wemos D1 Mini Pro 2.0 and Raspberry Zero
     PHAT DAC that has a PCM5102A mounted on it. Low cost for superb audio quality, as many other DACs it output
     up 192Khz 32 bits per sample, but ESP8266 I2S output can't output 24 or 32 bits? I do not tried infos on
     Arduino Core Documentation.

   Copyright (c) 2020 Massimo Meli. All right reserved.

   This example code is in the public domain, see attached Licence file
*/