A pre-history of the prediction of the amplitude of the second peak of the cosmic microwave background

In the previous post, I wrote about a candidate parent relativistic theory for MOND that could fit the acoustic power spectrum of the cosmic microwave background (CMB). That has been a long time coming, and probably is not the end of the road. There is a long and largely neglected history behind this, so let’s rewind a bit.

I became concerned about the viability of the dark matter paradigm in the mid-1990s. Up until that point, I was a True Believer, as much as anyone. Clearly, there had to be dark matter, specifically some kind of non-baryonic cold dark matter (CDM), and almost certainly a WIMP. Alternatives like MACHOs (numerous brown dwarfs) were obviously wrong (Big Bang Nucleosynthesis [BBN] taught us that there are not enough baryons), so microlensing experiments searching for them would make great variable star catalogs but had no chance of detecting dark matter. In short, I epitomized the impatient attitude against non-WIMP alternatives that persists throughout much of the community to this day.

It thus came as an enormous surprise that the only theory to successfully predict – in advance – our observations of low surface brightness galaxies was MOND. Dark matter, as we understood it at the time, predicted nothing of the sort. This made me angry.

How could it be so?

To a scientist, a surprising result is a sign to think again. Maybe we do not understand this thing we thought we understood. Is it merely a puzzle – some mistake in our understanding or implementation of our preferred theory? Or is it a genuine anomaly – an irrecoverable failure? How is it that a completely different theory can successfully predict something that my preferred theory did not?

In this period, I worked very hard to make things work out for CDM. It had to be so! Yet every time I thought I had found a solution, I realized that I had imposed an assumption that guaranteed the desired result. I created and rejected tautology after tautology. This process unintentionally foretold the next couple of decades of work in galaxy formation theory: I’ve watched others pursue the same failed ideas and false leads over and over and over again.

After months of pounding my head against the proverbial wall, I realized that if I was going to remain objective, I shouldn’t just be working on dark matter. I should also try just to see how things worked in MOND. Suddenly I found myself working much less hard. The things that made no sense in terms of dark matter tumbled straight out of MOND.

This concerned me gravely. Could we really be so wrong about something so important? I went around giving talks, expressing the problems about which I was concerned, and asking how it could be that MOND got so many novel predictions correct in advance if there was nothing to it.

Reactions varied. The first time I mentioned it in a brief talk at the Institute of Astronomy in Cambridge, friend and fellow postdoc Adi Nusser became visibly agitated. He bolted outside as soon as I was done, and I found him shortly later with a cigarette turned mostly to ash as if in one long draw. I asked him what he thought and he replied the he was “NOT HAPPY!” Neither was I. It made no sense.

I first spoke at length on the subject in a colloquium at the Department of Terrestrial Magnetism, where Vera Rubin worked, along with other astronomers and planetary scientists. I was concerned about how Vera would react, so I was exceedingly thorough, spending most of the time on the dark matter side of the issue. She reacted extremely well, as did the rest of the audience, many telling me it was the best talk they had heard in five years. (I have heard this many times since; apparently 5 years is some sort of default for a long time that is short of forever.)

Several months later, I gave the same talk at the University of Pennsylvania to an audience of mostly particle physicists and early-universe cosmologists. A rather different reaction ensued. One person shouted “WHAT HAVE YOU DONE WRONG!” It wasn’t a question.

These polar opposite reactions from different scientific audiences made me realize that sociology was playing a role. As I continued to give the talk to other groups, the pattern above repeated, with the reception being more positive the further an audience was from cosmology.

I started asking people what would concern them about the paradigm. What would falsify CDM? Sometimes this brought bemused answers, like that of Tad Pryor: “CDM has been falsified many times.” (This was in 1997, at which time CDM meant standard SCDM which was indeed pretty thoroughly falsified at that point: we were on the cusp of the transition to LCDM.) More often it met with befuddlement: “Why would you even ask that?” It was disappointing how often this was the answer, as a physical theory is only considered properly scientific if it is falsifiable. [All of the people who had this reaction agreed to that much: I often specifically asked.] The only thing that was clear was that most cosmologists couldn’t care less what galaxies did. Galaxies were small, non-linear entities, they argued… to the point that, as Martin Rees put it, “we shouldn’t be surprised at anything they do.”

I found this attitude to be less than satisfactory. However, I could see its origin. I only became aware of MOND because it reared its ugly head in my data. I had come face to face with the beast, and it shook my mostly deeply held scientific beliefs. Lacking this experience, it must have seemed to them like the proverbial boy crying wolf.

So, I started to ask cosmologists what would concern them. Again, most gave no answer; it was simply inconceivable to them that something could be fundamentally amiss. Among those who did answer, the most common refrain was “Well, if the CMB did something weird.” They never specified what they meant by this, so I set out to quantify what would be weird.

This was 1998. At that time, we knew the CMB existed (the original detection in the 1960s earning Penzias and Wilson a Nobel prize) and that there were temperature fluctuations on large scales at the level of one part in 100,000 (the long-overdue detection of said fluctuations by the COBE satellite earning Mathers and Smoot another Nobel prize). Other experiments were beginning to detect the fluctuations on finer angular scales, but nothing definitive was yet known about the locations and amplitudes of the peaks that were expected in the power spectrum. However, the data were improving rapidly, and an experiment called BOOMERanG was circulating around the polar vortex of Antartica. Daniel Eisenstein told me in a chance meeting that “The data are in the can.”

This made the issue of quantifying what was weird a pressing one. The best prediction is one that comes before the fact, totally blind to the data. But what was weird?

At the time, there was no flavor of relativistic MOND yet in existence. But we know that MOND is indistinguishable from Newton in the limit of high accelerations, and whatever theory contains MOND in the appropriate limit must also contain General Relativity. So perhaps the accelerations in the early universe when the CMB occurred were high enough that MOND effects did not yet occur. This isn’t necessarily the case, but making this ansatz was the only way to proceed at that time. Then it was just General Relativity with or without dark matter. That’s what was weird: no dark matter. So what difference did that make?

Using the then-standard code CMBFAST, I computed predictions for the power spectrum for two families models: LCDM and no-CDM. The parameters of LCDM were already well known at that time. There was even an imitation of the Great Debate about it between Turner and Peebles, though it was more consensus than debate. This enabled a proper prediction of what the power spectrum should be.

Most of the interest in cosmology then concerned the geometry of the universe. We had convinced ourselves that we had to bring back Lambda, but this made a strong prediction for the location of the first peak – a prediction that was confirmed by BOOMERanG in mid-2000.

The geometry on which most cosmologists were focused was neither here nor there to the problem I had set myself. I had no idea what the geometry of a MOND universe might be, and no way to predict the locations of the peaks in the power spectrum. I had to look for relative differences, and these proved not to be all that weird. The difference between LCDM and no-CDM was, in fact, rather subtle.

The main difference I found between models with and without dark matter was a difference in the amplitude of the second peak relative to the first. As I described last time, baryons act to damp the oscillations, while dark matter acts to drive them. Take away the dark matter and there is only damping, resulting in the second peak getting dragged down. The primary variable controlling the ratio of the first-to-second peak amplitude was the baryon fraction. Without dark matter, the baryon fraction is 1. In LCDM, it was then thought to be in the range 0.05 – 0.15. (The modern value is 0.16.)

This is the prediction I published in 1999:

the red lines in the left plot represent LCDM, the blue lines in the right plot no-CDM. The data that were available at the time I wrote the paper are plotted as the lengthy error bars. The location of the first peak had sorta been localized, but nothing was yet known about the amplitude of the second. Here was a clear, genuinely a priori prediction: for a given amplitude of the first peak, the amplitude of the second was smaller without CDM than with it.

Quantitatively, the ratio of the amplitude of the first to second peak was predicted to be in the range 1.54 – 1.83 for LCDM. This range represents the full range of plausible LCDM parameters as we knew them at the time, which as I noted above, we thought we knew very well. For the case of no-CDM, the predicted range was 2.22 – 2.57. In both cases, the range of variation was dominated by the uncertainty in the baryon density from BBN. While this allowed for a little play, the two hypotheses should be easily distinguishable, since the largest ratio possible in LCDM was clearly less than the smallest possible in no-CDM.

And that is as far as I am willing to write today. This is already a long post, so we’ll return to the results of this test in the future.