However, note that Graves et. The first traditional decoding strategy is best path decoding , which assumes that the most likely path corresponds to the most likely label. This is not necessarily true: suppose we have one path with probability 0. Best path decoding is fairly simple to compute; simply look at the most active output at every timestep, concatenate them, and convert them to a label via removing blanks and duplicates. Since at each step we choose the most active output, the resulting path is the most likely one.

As an alternative to the naive best path decoding method, we can perform a search in the label space using heuristics to guide our search and decide when to stop. One particular set of heuristics yields an algorithm called prefix search decoding , which is somewhat inspired by the forward-backward algorithm for hidden Markov models. The intuition behind prefix search decoding is that instead of searching among all labels, we can look at prefixes of strings.

We continue growing the prefixes by appending the most probable element until it is more probable that the prefix ends the string consists only of that prefix , at which point we stop. At each step, we maintain a list of growing prefixes. Initialise this list with a single element consisting of the empty prefix. Along with each prefix store its probability; we know that the empty prefix has probability one. Find the most likely prefix. Consider each possible extension of the prefix, or consider terminating the prefix and ending the string. If terminating the prefix has a higher probability than extending this or any other prefix, terminate the prefix; we have found our decoding.

If extending the prefix has a higher probability than terminating it, extend the prefix and store it with the new probability instead of the old, shorter prefix. Note that if given enough time, prefix search will find the true best decoding, and may thus require exponentially many prefixes. However, if the output distribution is concentrated around the best decoding, the search will finish significantly faster; also, heuristics may be used to speed it up.

For instance, Graves et. Note that at this point we have said nothing about how we can compute the probability of a prefix once we extend it, which is what we address next. To efficiently compute the probability of a prefix, we define some extra values which we will compute incrementally in a dynamic programming algorithm. With these in mind, we can proceed to implement the search algorithm described earlier.

## Diphone-based speech recognition using neural networks

First, we must initialize all our values. To understand the first of these, note that it's impossible that we see nothing when the path doesn't end in a blank since then we'd have seen that non-blank. Initialize both of these to the empty string as well. We now begin iteratively growing our prefixes, extending them by one character at a time. These equations are fairly intuitive. As we said before,. Now we have the probability of each extended prefix and the probability of ending with each one.

When you compute these for a prefix, do the following:. We wish to continue growing our prefix until the current estimate of the best labeling has higher probability than any of our other options from the best prefix. After each step, we have these values, so we can easily test for termination. This entire algorithm is summarized in the following graphic, taken from Alex Graves' dissertation :.

- Speech recognition using neural networks - IEEE Conference Publication;
- Lecture Notes in Microeconomic Theory: The Economic Agent;
- Natural Product Biosynthesis by Microorganisms and Plants, Part A.
- Speech-to-Text using Convolutional Neural Networks.

Now that we have defined the probability distribution used in CTC networks as well as figured out how to decode the CTC network output, we are left with the question of how we can train our CTC networks. In order to train our network, we need an objective function, which we can then minimize via some standard minimization algorithm such as gradient descent or hessian free optimization. In this section, we derive this objective function. The objective function is based off maximum likelihood; minimizing the objective function maximizes the log likelihood of observing our desired label. Naively computing this is computationally intractable, as demonstrated via the following equation which we came up with above :.

However, we can compute this efficiently via a dynamic programming algorithm similar to the one we used to do decoding. This algorithm, however, has a forward and backward pass.

## Convolutional Neural Networks for Raw Speech Recognition

The forward pass computes probabilities of prefixes, and the backward pass computes probabilities of suffixes. The maximum likelihood function works by probabilistically matching elements of the label sequence with elements of the output sequence. We know that the output sequence will have many blanks; in particular, we expect that there will very often be a blank between successive letters.

To simplify our matching, we can account for this by adjusting our mental model of the label we're matching. This way, if the network outputs blanks between its letters, they will correspond to existing blanks between the letters in the label. This forms a base case. Note that the sum of this case is identical to that of the previous case. We can now formulate our objective function. Now that we have an objective function, can devise a training algorithm to minimize it. As we'll see, this is where the backwards pass of our forward-backward algorithm comes into play.

We minimize it by taking the gradient with respect to the weights, at which point we can use gradient descent. Note also that since all training samples are independent, we will just compute our derivatives for a single training sample; simply sum over all training samples to deal with the entire dataset.

In order to compute our gradients, we are going to need our set of backwards variables. We also know that it's impossible to see a two or more character suffix if we're only looking at the last time output:. Since each term is for a distinct prefix or suffix, the cross product of these two sets yields all possible prefixes and suffixes.

When we multiply two terms both of which are products over the path from the two sums, the resulting term is also just a product over the path. This yields the equation. Thus, we can write that. Then, we can write the derivative as. Recall that the final objective function is actually the natural log of the probability.

However, we know that. This concludes our analysis of connectionist temporal classification CTC networks; the details may be accessed in this paper and Alex Graves' dissertation , both of which address several other issues which arise in practice with CTC networks, and include experimental findings related to their use. The connectionist temporal classification model we described above does a good job as an acoustic model; that is, it can be trained to predict the output phonemes based on the input sound data.

However, it does not account for the fact that the output is actually human language, and not just a stream of phonemes. We can augment the acoustic model with a "linguistic" model, one that depends solely on the character stream, and not on the sound data. A full account of it may be viewed in this paper. Using the same architecture as we defined in the first section before we looked into CTC networks , we train an RNN to do one-step prediction.

Now we have two models - one RNN that does character-level prediction, and one that does sound-based prediction. The length of the vector is dependent on the number of characters in the alphabet, with potentially an extra space for the blank. Note that these functions are effectively predicting the next character emitted. These have a similar justification as in the previous section. Next, we proceed through the rest of the CTC algorithm in a similarly motivated way.

Decoding, however, must be done with a beam search, which again is documented in the original paper. Finally, we have all the components we need to create our final network.

### chapter and author info

Our final network greatly resembles the RNN tranducer network we discussed above. While that is the standard formulation, Graves et.

- Suspensions of Colloidal Particles and Aggregates.
- Recognizing Speech Commands Using Recurrent Neural Networks with Attention.
- Finding and revealing your sexual self: a guide to communicating about sex.
- Aufgabensammlung Technische Mechanik [Hauptbd.]. Mit 939 Aufgaben.
- Future technology: Voice Recognition Using Neural Networks.
- Future technology: Voice Recognition Using Neural Networks.
- Simple Audio Recognition!

Note that the function. Instead, Graves et. They find that this decreases deletion errors during speech recognition. Sreenivasa Rao, Partha Pratim Das. N, Kishor K. S, Sri Rama Murty Kodukula. Harald Baayen. Rao, Harinath Garudadri. Kelley, Benjamin V. Who Said That? Nixon, Tomas O.

Louis ten Bosch, Lou Boves. Bepari, Joyanta Basu. Ji, Kirrie Ballard. Madhavi, Hemant Patil. Pandey, K S Nataraj. P, Govind D. Who Are You Listening to? Should Code-switching Models Be Asymmetric? Barbara E. Bulut, Lakshmish Kaushik, Chengzhu Yu. Meltzner, Rupal Patel. Lightly Supervised vs. Sreenivasa Rao, Sanjay Kumar Gupta. Patil, Madhu Kamble, Hemant Patil. How Did You like ? Thiagarajan, Visar Berisha, Andreas Spanias. Foltz, Alex S.

Cohen, Terje B. Wong, Hanjun Liu, Francis C. Speech Segments and Voice Quality. Acoustic Analysis-Synthesis of Speech Disorders. Deception, Personality, and Culture Attribute. Spoken Dialogue Systems and Conversational Analysis.