playing ogg voices as smooth as possible in java TTS

[Nawras Nzar]

hello guys
Am developing a text-to-speech for my own language in java (its my final project which is not developed before, therefor I could NOT use built-in classes). I can recognize the diphones for input text, for playing: I place the diphones in an array, after the input text analysis completed I play the audio files (which are in ogg format) according to the diphones in the array one by one. what I want to ask is: what do you think about this way for playing separate diphones? and what to do to make it smoother because its really not smooth (there is a big gap between playing one file and next one)

[Stephan van Hulst]

Hi, welcome to CodeRanch!

Please UseRealWords. I imagine not many people know what the abbreviation TTS stands for, so it's better to just write text-to-speech.

As for your problem, I haven't really worked with audio in Java that much, but I imagine that instead of playing separate files, it's better to buffer the files well before they're about to be played, or create an entire new file consisting of the phones and diphones for the part of speech that you want to play.

[Nawras Nzar]

thanks for reply...
I don't think it's possible to merge all files, but buffering them well maybe.
but I'm seeking for better approach
thank you again.

[Stephan van Hulst]

A better approach than buffering? What's wrong with buffering?

[Nawras Nzar]

sorry I didn't mean it's not good
but I personally didn't tried it before so I don't know how much effective it will be.
my audio files are in a package for example sounds and I just give the path and play
better say I don't know how exactly to buffer them!!!

[Stephan van Hulst]

Exactly, that's the problem. You're loading the files one at a time, so there's a gap between two fragments when the program has to load the next file. If the sounds are buffered, they can start playing immediately after the previous sound ends.

Now, if you show us how you're currently playing the sound fragments, maybe we can show you how to improve smooth playback.

[Nawras Nzar]

    public void play() {
        // Now get a line & use it to read the audio file;
        if (line != null) {
            try {
                if (!line.isOpen()) {
                    line.open(decodedFormat);
                }
            } catch (LineUnavailableException ex) {
                Logger.getLogger(Sound.class.getName()).log(Level.SEVERE, null, ex);
            }

            line.start();

            line.write(audioDataArray, 0, audioDataArray.length);
            // Stop
            //we kill so much time here in drain
            line.drain();
            line.stop();
            line.close();
        }

if I remove line.drain(); it become very much faster, but I don't know if any drawbacks I will face

[Stephan van Hulst]

This is very little to go on. I don't know how you're loading files, how you're handling your fragment queuing, what types you're using, etc.

Please post an SSCCE.

[Nawras Nzar]

from this class tokens send to Sound.java class

Speech.java

package text2speech.speech;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.sound.sampled.LineUnavailableException;
import text2speech.parser.LanguageDescriptor;
import text2speech.parser.TextTokenizer;
import text2speech.parser.TokenElement;

public class Speech {

    private String voice;
    private TextTokenizer tokens;
    Map<TokenElement, Sound> soundMap;
    
    public Speech(TextTokenizer tokens, String voice) {
        this.tokens = tokens;
        this.voice = voice;
        soundMap = new HashMap<TokenElement, Sound>();
//        loadSounds();
    }

    public void speak(LanguageDescriptor descriptor) throws LineUnavailableException, IOException {
        Sound sound;
        for (TokenElement token : tokens.getTokenList()) {
            sound = descriptor.getTokenSounds().get(token);
            if(sound!=null){                
                sound.play();
            }
        }
    }

    private void loadSounds() {
        for (TokenElement token : tokens.getTokenList()) {
            loadSound(token);
        }
    }

    private void loadSound(TokenElement token) {
        if (!soundMap.containsKey(token)) {
//            soundMap.put(token, new Sound(token.getSound(voice)));
        }
    }
}

this is Sound.java

package text2speech.speech;

import java.io.ByteArrayInputStream;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.SourceDataLine;
import javax.sound.sampled.UnsupportedAudioFileException;
import text2speech.Main;
import text2speech.parser.LanguageDescriptor;

public class Sound {

    private byte[] audioDataArray;
    private AudioFormat decodedFormat;
    private SourceDataLine line;

    public Sound(String filename, LanguageDescriptor languageDescriptor) {
        // Construct a byte array of the length of the file (with header)
        byte[] soundReadBuffer = new byte[(int) getFile(filename,
                languageDescriptor).length()];
        AudioInputStream soundInputStream = null;

        // Get a data input stream from the audio file
        DataInputStream dataInputStream = new DataInputStream(
                getFileAsStream(filename, languageDescriptor));

        // Read all the audio file bytes (with headers) to a byte array
        // Now the sound resides in memory & later we will read it from there.
        try {
            dataInputStream.readFully(soundReadBuffer);
        } catch (IOException ex) {
            Logger.getLogger(Sound.class.getName()).log(Level.SEVERE, null, ex);
        }

        // Obtain a byte input stream from the data input stream, this
        // Will be used as an input stream instead of the file on disk, hoping
        // That this will minimize delays.

        ByteArrayInputStream byteStream = new ByteArrayInputStream(soundReadBuffer);
        try {
            dataInputStream.close();
        } catch (IOException ex) {
            Logger.getLogger(Sound.class.getName()).log(Level.SEVERE, null, ex);
        }

        // Get an audio input stream from the audio system, using the byte
        // input stream we constructed.
        // Now we read directly from the line whose contents are in memory!

        AudioInputStream in = null;
        AudioFormat baseFormat = null;
        try {
            in = AudioSystem.getAudioInputStream(byteStream);
        } catch (UnsupportedAudioFileException ex) {
            Logger.getLogger(Sound.class.getName()).log(Level.SEVERE, null, ex);
        } catch (IOException ex) {
            Logger.getLogger(Sound.class.getName()).log(Level.SEVERE, null, ex);
        }
        if (in != null) {
            baseFormat = in.getFormat();
            decodedFormat = new AudioFormat(
                    AudioFormat.Encoding.PCM_SIGNED,
                    baseFormat.getSampleRate(), 16, baseFormat.getChannels(), baseFormat.getChannels() * 2, baseFormat.getSampleRate(), false);
            // Get AudioInputStream that will be decoded by underlying VorbisSPI
            soundInputStream = AudioSystem.getAudioInputStream(decodedFormat, in);
        }

        // Get the line to be used for playing back.
        DataLine.Info info = new DataLine.Info(SourceDataLine.class, decodedFormat);
        try {
            line = (SourceDataLine) AudioSystem.getLine(info);
        } catch (LineUnavailableException ex) {
            Logger.getLogger(Sound.class.getName()).log(Level.SEVERE, null, ex);
        }

        // Load the data
        loadData(soundReadBuffer, soundInputStream);
    }

    private void loadData(byte[] soundReadBuffer, 
            AudioInputStream soundInputStream) {
        
        // TODO since the size of audio data in bytes is not known at this point,
        // We first copy to an array list, & because a line only accepts 
        // byte arrays of data, we have to copy the array list elements one by
        // one to a byte array. We need to try to find a more elegant way of 
        // doing this.
        ArrayList<Byte> audioData = new ArrayList<Byte>();
        int nBytesRead = 0;

        while (nBytesRead != -1) {
            try {
                nBytesRead = soundInputStream.read(soundReadBuffer);
                for (int i = 0; i < nBytesRead; i++) {
                    audioData.add(soundReadBuffer[i]);
                }
                audioDataArray = new byte[audioData.size()];
                 for(int i = 0; i< audioData.size(); i++)
                    audioDataArray[i] = audioData.get(i);
            } catch (IOException ex) {
                Logger.getLogger(Sound.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
    }

    public void play() {
        // Now get a line & use it to read the audio file;
        if (line != null) {
            try {
                if (!line.isOpen()) {
                    line.open(decodedFormat);
                }
            } catch (LineUnavailableException ex) {
                Logger.getLogger(Sound.class.getName()).log(Level.SEVERE, null, ex);
            }

            line.start();

            line.write(audioDataArray, 0, audioDataArray.length);
            // Stop
//if comment line.drain(); playing will be faster without gaps

            line.drain();
            line.stop();
            line.close();
        }
    }

    /**
     * This method takes care to find the proper path of the desired sound. By
     * making sure that it will use the Resource loader if the sound is inside
     * the distributed JAR file
     * @param filename the sound name to be loaded, in
     * this format: sound/sound_name.ogg
     * @param languageDescriptor The language descriptor
     * @return The correct path of the sound
     */
    private File getFile(String filename,
            LanguageDescriptor languageDescriptor) {
        File path;
        if (!languageDescriptor.isExternal()) {
            path = new File(Main.class.getResource("resources"
                    + System.getProperty("file.separator")
                    + languageDescriptor.getDescriptorPath()
                    + System.getProperty("file.separator")
                    + filename).getFile());
        } else {
            path = new File(languageDescriptor.getDescriptorPath()
                    + System.getProperty("file.separator")
                    + filename);
            path = null;
            System.exit(1);

        }
        return path;
    }

    private InputStream getFileAsStream(String filename,
            LanguageDescriptor languageDescriptor) {
        InputStream path;
        if (!languageDescriptor.isExternal()) {
            path = Main.class.getResourceAsStream("resources"
                    + System.getProperty("file.separator")
                    + languageDescriptor.getDescriptorPath()
                    + System.getProperty("file.separator")
                    + filename);
        } else {
            try {
                path = new FileInputStream(languageDescriptor.getDescriptorPath()
                        + System.getProperty("file.separator")
                        + filename);
            } catch (FileNotFoundException fnfex) {
                Logger.getLogger(Sound.class.getName()).log(Level.SEVERE,
                        "{0} could not be found\n" + fnfex.getMessage(),
                        filename);

                path = null;
                System.exit(1);

            }
        }
        return path;
    }
}

[Nawras Nzar]

Mr.Stephan van Hulst are you there?
I really need help...