Subjective Model Predicts Human Brain's Response to Timbral Differences Better Than Objective Models

A quick perusal of the article left me wondering how they constructed these sounds that represented the various models. In general the sounds constructed utilizing subjective descriptions more accurately predicted mapped areas in the brain as determined on fMRI than did sounds constructed using measured values of the stimulus or its downstream representation in the cochlear. Top down processing of complex subjective musical phenomena like timbre likely better explains their perception if judged by fMRI responses 

6 hours ago, pkane2001 said:

 

You seem to read too much into this paper. It was an evaluation of  computer-based neural network performance trained on a set of subjective descriptors as input, and neuron activation patterns as output. Other models were similarly trained using spectrum centroid and spectrum-temporal inputs instead of subjective descriptors. Again, only COMPUTER-based neural network performance was compared in this study, not the human brain.

 

Spectrum-temporal model was within the margin of error of timbre model in predicting activation patterns:
 

 

There is nothing in the paper to conclude that one set of inputs is used by the brain to recognize timber over any others. The only conclusion is that a computer-based neural net trained on subjective descriptors as input was better at predicting cortex activation patterns than a couple of other neural nets trained on different set of inputs (cochlear mean and spectral centroid), while the STM model performed just as well as Timber in this prediction task.

 

The result is actually interesting, but doesn't justify any of the conclusions you seem to draw from it related to audio.

Hi Paul,

I originally missed the detail of  how they generated the audio samples from the 42 natural orchestral instrument stimuli. If correct, they used Matlab (Mathworks) to analyze the audio and classify into different models that they then generated futher audio samples for training and testing. Would that be correct (I have only come across matlab image tools before)?

Anyway I can relate if you have some reservations about this brave new world of machine neural networks but in all honesty I would have thought it was right up your alley, machines analyzing and outputting audio samples with Trump-like "HUGE" accuracy, objective measurements and all that jazz. They even have a Delta Audio Matlab!

I believe @AcousticTheory was actually just relaying the findings and conclusions in the study and its referenced source material. What do you have an issue with exactly and how would you have done things differently?

14 hours ago, pkane2001 said:

 

Hi David,

 

What I disagree with is the conclusion by @AcousticTheory that this study somehow proves anything about how human brain processes timbre. There was no such analysis done. The goal of the study was to test a computer model that predicts neuron activation patterns using computer-generated neural networks with different types of inputs, not human brain. To draw any conclusions from this study as to how we humans process timbre is not supported by the facts.

14 hours ago, pkane2001 said:

 

It is a study designed to create a computer model predicting neural pattern activation. That is the goal. To do this, human brain(s) are required to collect information that can be used to train and then test the model.

 

Paul I think we need to accept what the aims or goal/s that were being explored as is stated in the study, the method employed notwithstanding, or whether you feel such goals were achieved, or conclusions valid - but your comments do raise some interesting points (separate post below).

Firstly their goal of the study is as stated by them (and which they conclude supported by the study) as quoted from different parts of the article:

"Here we test an encoding model that is based on five subjectively derived dimensions of timbre to predict cortical responses to natural orchestral sounds. [other researchers] consistently found [these] five [subjective] dimensions to be both necessary and sufficient for describing the timbre space of these orchestral sounds. The aim of the current study was to determine whether similar dimensions can be identified in the cortical representations of timbral differences." and they "explored the possibility that a subjectively based model of timbre could predict patterns of cortical activation in response to sound." This was done compared to other models that, although I don't recall as stated specifically, could be held as more "objective" models compared to the "subjective model of timbre"

Secondly, Their conclusions:

"The timbre model provides an efficient representation of processing in human auditory cortex via a compact model whose features are based on subjective ratings of timbre. Our results suggest that the distributed neural representation of timbre in the cortex may align with perceptual categorizations of timbre. Consequently, it may be possible to assign semantic labels to the multidimensional tuning of neuronal populations"

Thirdly, Their caveats:

First is the universal caveat that we need more studies.

"Since the employed timbre model was customized for this particular set of orchestral instruments, studies that test a broader range of stimuli (i.e., more musical instruments, speech, and other natural sounds) are recommended in order to determine the extent of this model’s generalizability."

then

"an area that warrants future research is the development of methods to optimally combine models that explain different parts of the variance (see e.g., de Heer et al., 2017)."

14 hours ago, pkane2001 said:

 

It is a study designed to create a computer model predicting neural pattern activation. That is the goal. To do this, human brain(s) are required to collect information that can be used to train and then test the model.

 

So, following on from their goals as stated by the authors, it was not as you say “designed to create a computer model predicting neural pattern activation”. The computer model here is the test method used by which they are testing various hypotheses or “goals”. Some may find objection to the method they employed. That objection being to conclusions relating to as you say “proves anything about how human brain processes timbre”.

I do find that interesting. Two points spring to mind.

They are assuming that the test method used ie models generated by real sounds once processed by a machine neural network (AI to most?) and then the same machine used to generate real sounds representing the various models used in testing (if I have that right)– is a valid method. This reminds me of other situations where I wish to distinguish the actual validity of the test method procedure, not just assuming it is valid eg ABX blind testing procedures and why there might be false negatives. I will not be drawn into an argument about blind testing except to say no-one has provided to my satisfaction real test of test measurements as for example, its positive and negative predictive values expressed as percentage probabilities.

It also raises the issue of how various audio measurements are sometimes held to represent what we hear. This study measures real sound, compares signals and concludes something about those signals to be same or different and in this case outputs real sound based on measurements and analysis by this machine “neural network”. I do see at least parallels to what other audio test devices do. In this case , using objective measures (and using their conclusions) favors a subjective outcome when it comes to the perception of timbre.

10 hours ago, PeterG said:

 

Very interesting, and a reasonable explanation for why, just for example, a tube based amplifier may produce a more realistic sounding acoustic guitar or vocal than a solid state with better measurements

One wonders whether objectively derived neural networks based on subjective perceptual models may be a much more realistic way to assess the performance of audio gear when it comes to matching what we hear.

1 hour ago, pkane2001 said:

Sorry, couldn't answer in detail earlier (and still really can't from this d*mn tiny screen), but neural networks are hardly an objective way to measure anything. Training them, and selecting the right inputs as well as selecting training and testing data is an art rather than science. What's more, there is no guarantee these will make as accurate a prediction on a wider data set, for example one that includes the same exact piece played through two different amplifiers.

 

Accuracy of the timbre model (63%, +/-1%) to predict a brain activation pattern, while may be a few percent better than the previous, competing STM model (60%, +/-1%), is still very low and not a major improvement, IMHO. Also remember, this is while trying to differentiate between completely different, orchestral recordings. 

 

vv

1 hour ago, pkane2001 said:

neural networks are hardly an objective way to measure anything. Training them, and selecting the right inputs as well as selecting training and testing data is an art rather than science. What's more, there is no guarantee these will make as accurate a prediction on a wider data set, for example one that includes the same exact piece played through two different amplifiers.

Perhaps the word "measurement" is out of context relating to ANN which seem to talk about "input signals" or nodes which depending on the model get somehow 'measured'/classified in some fashion. A grayscale image may have nodes representing the level of each pixel and 'measuring' tens of thousands of input nodes.

The important thing for me however is their predictive value in "outputs" especially in the area of so called "semantic segmentation" in complex nonlinear functions such as perception of timbre qualities might be.

Those predictions can be objectively measured in various ways, the most obvious is accuracy, how often does it get it right. Both an objective and meaningful figure.

Traditional audio measurements do not always correlate to perceptual predictions.

Even more interesting ,as one person with a Bachelors of Engineering in Artificial Intelligence & Robotics put it when measuring the learning performance of ANN's there is also a matrix that "breaks down true positives, true negatives, false positives, and false negatives, giving a clearer picture, especially in binary classification tasks"

2 hours ago, pkane2001 said:

Accuracy of the timbre model (63%, +/-1%) to predict a brain activation pattern, while may be a few percent better than the previous, competing STM model (60%, +/-1%), is still very low and not a major improvement, IMHO. Also remember, this is while trying to differentiate between completely different, orchestral recordings. 

I do think you may have been a little dismissive of this study based on your previous comments regarding the study's stated goals and their results/conclusions (see my previous posts and from @AcousticTheory).

You correctly point out one results in isolation to another important result according to the authors. The subjective timbre model does more accurately predict the mapping to some specific auditory cortical locations “performing significantly better” than the spectrotemoral model and suggesting it is “capturing some semantic or perceptual tuning properties of the auditory cortex that extend beyond those captured by the spectrotemporal model”.

That is a finding related to how humans process timbre. “The timbre model provides an efficient representation of processing in human auditory cortex via a compact model whose features are based on subjective ratings of timbre. Our results suggest that the distributed neural representation of timbre in the cortex may align with perceptual categorizations of timbre. Consequently, it may be possible to assign semantic labels to the multidimensional tuning of neuronal populations. “

12 hours ago, pkane2001 said:

A more interesting paper (IMHO) than the one in the OP is this:
 

Acoustic structure of the five perceptual dimensions of timbre in orchestral instrument tones

Taffeta M. Elliott,a) Liberty S. Hamilton, and Frederic E. Theunissen [https://doi.org/10.1121/1.4770244]

 

The five dimensional timbre model in this paper is what is used to provide "subjective" inputs to the neural net in the brain activation study.

 Yes I did have a look previously as it was referenced quite a bit within this present study and indeed this study builds on that work comparing models of temporal and spectral characteristics. That I think is the point of this study making it so interesting.