Contents

• Introduction
• Downloading and compiling
• Running chains and input parameters
  • Input Parameters
  • Output Files
• Analysing samples and plotting
  • GetDist Parameters
  • GetDist Output
  • GetDist Plotting
• Convergence statistics
• Parameterizations
• Data
• Programming
• Version history
• References
• FAQ

On a multi-processor machine you can start to get good results in a couple of hours. On single processors you'll need to set aside rather longer. You can also run on a cluster. Also check our chains page to see if we have suitable runs available that you can importance sample from very quickly (typically just seconds to re-compute a few thousand likelihoods).

By default CosmoMC uses a simple Metropolis algorithm, but there are options for slice sampling and more powerful methods of exploring the fast/slow parameter space. The program takes as inputs estimates of central values and posterior uncertainties of the various parameters. The proposal density can use information about parameter correlations from a supplied covariance matrix: use one if possible as it will significantly improve performance. There is an option to esimate the covariance about the best-fit point, and one covariance matrix is supplied for the default base parameters. If you compile and run with MPI (to run across nodes in a cluster), there is an option to dynamically learn the proposal matrix from the covariance of the post-burn-in samples so far. The MPI option also allows you to terminate the computation automatically when a particular convergence criterion is matched. MPI is recommended.

There are two programs supplied cosmomc and getdist. The first does the actual Monte-Carlo and produces sets of .txt chain files and (optionally) .data output files (the binary .data files include the theoretical CMB power spectra etc.). The "getdist" program analyses the .txt files calculating statistics and outputs files for the requested 1D, 2D and 3D plots (and could be used independently of the main cosmomc program). The "cosmomc" program also does post processing on .data files, for example doing importance sampling with new data.

Please e-mail details of any bugs or enhancements to Antony Lewis. If you have any questions please ask in the CosmoCoffee computers and software forum. You can also read answers to other people's questions there.

Downloading and Compiling

• Get the download and unzip and untar it (run "gunzip cosmomc.tar.gz", then "tar -xf cosmomc.tar")
• To use WMAP3 you also need Healpix and CFITSIO installed
• For WMAP 3-year (obviously skip the first two steps below if already installed somewhere)
  • Download the WMAP likelihood code and data from here (get wmap_likelihoodcode_sw_only_v2p1.tar.gz or later).
  • Extract to some separate directory
  • Edit WMAP_3yr_options.f90 to hard code the full path instead of ./data/
  • Make and run the test program supplied
• Uncomment the relevant top parts of the Makefile in the camb directory depending on what system you are using, and whether or not you want to use OpenMP.
• Run "make all" in the camb subdirectory
• In cosmomc's source subdirectory edit the Makefile for your cfitsio, healpix and WMAP3 directories and compiler. Uncomment the relevant parts of the Makefile depending on what system you are using and whether or not you want to use MPI and/or OpenMP.

  Using MPI simplifies running several chains and proposal optimization. MPI can be used with OpenMP: generally you want to use OpenMP to use all the shared-memory processors on each node of a cluster, and MPI to run multiple chains on different nodes (the program can also just be run on a single CPU). The intel compilers are changing frequently, there is a file Makefile_intel supplied you can use with the 7.1 versions that give problems.

• Run "make all" in the source subdirectory (try "gmake all" if you get an error)

You can download the Intel f90 compiler for Linux here (earlier versions of ifc 8 did not work; get Build 20040716Z or later. You will also need MKL or some other LAPACK library installation). There is also now a GNU F95 compiler you could try. Note that version 7.4 of the SGI compiler is buggy - install 7.4.1m compilers plus patch 5292 [or comment out "flush" in utils.F90]. Please let me know if you have specific fixes for other compilers.

Using (Compaq) Visual Fortran there's no need to use the Makefile, just open cosmomc.dsw in the source folder, and set params.ini as the program argument under Project, Settings, Debug and set the working directory to ..\. Under Tools, Options, Directories set the include path to [cxml path]/CXML/INCLUDE and lib path to [cxml path]/CXML/LIB. Don't install the 6.6C3 update as it gives compiler errors (6.6B is fine). You then need to add healpix and cfitsio files to your project depending on where they are on your system.

Note that to compile cosmomc you need to link to LAPACK (for doing matrix diagonalization, etc) - you may need to edit the Makefile to specify where this on your system.

To change the l_max which is used for generating Cls you'll need to edit the value in cmbtypes.f90, run "make clean" then "make" to rebuild everything. Note l_max should be 50 larger than the largest l which you need accurately. You can also change matter_power_lnzsteps, the number of redshifts at which matter power spectra are sampled.

The default code includes polarization. You can edit the num_cls parameter in cmbtypes.f90 to include just temperature (num_cls=1), TT, TE and EE (num_cls=3) or TT, TE, EE and BB (num_cls=4). You will need the last option if you are including tensors and using polarized data. You can use temperature-only datasets with num_cls 3 or 4, the only disadvantage being that it will be marginally slower, and the .data files with be substantially larger. For WMAP data you need num_cls = 3 or 4.

Running and input parameters

See the supplied params.ini file for a fairly self-explanatory list of input parameters and options. The file_root entry gives the root name for all files produced. Running using MPI on a cluster is recommended if possible as you can automatically handle convergence testing and stopping.

• Running individual chains
  
  Run the program using

  ./cosmomc params.ini

  The samples will be in file_root.txt, etc. You can start several instances of the program generating separate chains using

  ./cosmomc params.ini 1
  ./cosmomc params.ini 2
  etc.

  In this case samples will be in the files file_root_NN.txt, where NN labels the chain number.
  
  
• MPI runs on a cluster
  
  If you edit the Makefile to run with MPI (compile with -DMPI), a number of chains will be run according to the number of nodes assigned by MPI when you run the program. Output file names will have "_1","_2" appended to the root name automatically. You may be able to run over 4 nodes using e.g.

  mpirun -np 4 ./cosmomc params.ini

  There is also a supplied perl script runMPI.pl that you may be able to adapt for submitting jobs to a PBS queue, running lamboot, etc, e.g.

  perl runMPI.pl params 4

  to run 4 chains over four nodes using the params.ini parameters file (the script is set up by default for the CITA cluster - edit ppn=2 to the number of CPUs per node you have). On the UK Cosmos computer see the FAQ below.
  
  In the parameters file you can set MPI_Converge_Stop to a convergence criterion (see Convergence Diagnostics. Small numbers are better convergence; generally need R-1 < 0.1, but may want much smaller especially for importance sampling or accurate confidence limits. You can also directly impose an error limit on confidence values - see the parameter input file for details). If you set MPI_LearnPropose = T, the proposal density will be updated using the covariance matrix of the last half of all samples (across chains) so far.
  
  Using MPI_LearnPropose will significantly improve performance if you are adding new parameters for which you don't have a pre-computed covariance matrix. However adjusting the proposal density is not strictly Markov, though asymptotically it is as the covariance will converge. The burn in period should therefore be taken to be larger when learning the proposal density to allow for the time it takes the covariance matrix to converge sufficiently (though in practice it probably doesn't matter much in most cases). Note also that as the proposal density changes the number of steps between independent samples is not constant (i.e. the correlation length should decrease along the chain until the covariance has converged). The stopping criterion uses the last half of the samples in each chain, minus a (shortish) initial burn in period. If you don't have a starting covariance matrix a fairly safe thing to do is set ignore_rows=0.5 in the GetDist parameter file to skip the first half of each chain.
  
  You can also set the MPI_R_StopProposeUpdate parameter to stop updating the proposal density after it has converged to R-1 smaller than a specified number. After this point the chain will be strictly Markovian. The number of burn in rows can be calculated by looking at the params.log file for the number of output lines at which R-1 has reached the required value.

  If things go wrong check the .log and any error files in your cosmomc/scripts directory.

• Parameter limits and proposal density
  The parameter limits, estimated distribution widths and starting points are listed as the paramxxx variables. The proposal density changes parameters using a Gaussian with a standard deviation given by the specified width multiplied by the value of the propose_scale parameter. The widths should be of the order of the posterior width unless the parameter is very correlated in which case it should be smaller. If you specify a propose_matrix (approximate covariance matrix for the parameters), the parameter distribution widths are determined from its eigenvalues instead, and the proposal density changes the parameter eigenvectors. The covariance matrix can be computed using "getdist" once you have done one run - the file_root.covmat file. A .covmat file is supplied for the basic 6-parameter set. The covariance matrix does not have to include all the parameters that are used - zero entries will be updated from the input propose widths of new parameters. The "start width" paramxxx entry determines the randomly chosen dispersion of the starting position about the given centre, and should be as large or larger than the posterior (for the convergence diagnostic to be reliable, chains should start at widely separated points). The scale of the proposal density relative to the covariance is given by the propose_scale parameter. If your propose_matrix is significantly broader than the expected posterior, this number can be decreased.
  If you don't have a propose_matrix, set estimate_propose_matrix = T to automatically esimate it by numerical differentiation about the best fit point. To use this option you should not have parameter priors cutting off the distribution near the maximum likelihood.
  
  
• Sampling method
  Set sampling_method=1 to use the default Metropolis algorithm in the optimal fast/slow subspaces. Other options are slice sampling (2), slice sampling fast paramters (3), and directional gridding (4). See the notes for a more detailed explanation.
  
  
• Best-fit point
  Set action =2 to just calculate the best-fit point and stop. Set delta_loglike to the tolerance on the log likelihood for finding the best fit. The values are output to a file called file_root.minimum.
  
  
• Data file output
  Since consecutive points are correlated and the output files can get quite large, you may want to thin the chains automatically: set the indep_sample parameter to determine at what frequency full information is dumped to a binary .data file (which includes the Cls, matter power spectrum, parameters, etc). If zero no .data file is generated. You only need to keep nearly uncorrelated samples for later importance sampling. You can specify a burn_in, though you may prefer to set this to zero and remove entries when you come to process the output samples.
  
  
• Threads and run-time adjustment
  The num_threads parameter will determine the number of openMP threads (in MPI runs, usually set to the number of CPUs on each node). Scaling is linear up to about 8 processors on most systems, then falls off slowly. It is probably best to run several chains on 8 or fewer processors. You can terminate a run before it finishes by creating a file called file_root_NN.read containing "exit =1" - it will read it in, delete the .read file and stop. The .read file can also have "num_threads =xx" to change the number of threads (per chain) dynamically. If you have multiple chains you can create file_root.read which will be read by all the chains. In this case the .read file is not deleted automatically.
  
  
• Post-processing (e.g. importance sampling)
  The action variable determines what to do. Use action=1 to process a .data file produced by a previous MCMC run - this is used for importance sampling with new data, correcting results generated by approximate means, or re-calculating the theoretical predictions in more detail. If action=1 set the redo_xxx parameters to determine what to do. With redo_add=F you should include all the data you want used for the final result (e.g. if you generated original samples with CMB data, also include CMB data when you importance sample - it will not be weighted twice). With redo_add=T you should include only data you want to add - the new likelihoods are added to those already stored in the chains. PostProcessing is fastest if you have binary .data files from an original run, however if you do not have these you can use redo_from_text=T to read in chains from .txt files.
  
  
• Exploring tails and high-significant limits
  The temperature setting allows you to sample from P^(1/T) rather than P - this is good for exploring the tails of distributions, discovering other local minima, and for getting more robust high-confidence error bars.
  
  
• Checkpointing
  Set checkpoint = T in the .ini file to checkpoint: i.e. generate files so that if the processes are terminated they can be restarted again from close to where they left off. With MPI this setting creates file_root.chk files used to store the current status. To continue a prematurely terminated run, just run again using exactly the same commands and files as originally. Note that if you use checkpointing, but you want to overwrite existing chains of the same name, you must manually delete all the file_root.* files (otherwise it will attempt to continue from where they left off). Please note this feature is not fantastically well tested, please let us know of any problems. Also note that for checkpointing to work the first run must run for at least past the initial burn phase (you can see when the file_root.chk files have been produced), otherwise the run will just start from scratch again.

    weight like param1 param2 param3 ...

The weight gives the number of samples (or importance weight) with these parameters. like gives -log(likelihood). The getdist program could be used completely independently of the cosmomc program.

Run getdist distparams.ini to process the chains specified in the parameter input file distparams.ini. This should be fairly self-explanatory, in the same format as the cosmomc parameter input file. Note that sigma_8 is only computed if you are including LSS data when generating the chain (as computing the matter power spectrum slows things down considerably; You can post-process to compute sigma8 if you like, see action=1 in the cosmomc input file).

GetDist Parameters

• GetDist processes the file_root.txt file (or, if there are multiple chains, set the chain_num parameter), and outputs statistics, marginalized plots, samples plots, and performs PCA (principle component analysis).
  
  
• Set ignore_rows to a positive integer to ignore that number of output rows as burn-in, or to a fraction <1 to skip that fraction of each chain's rows.
  
  
• Set the thin_factor parameter to produce a file_root_thin.txt file containing every thin_factorth sample.
  
  
• Set adjust_priors to adjust the sample multiplicities for a new prior (write the corresponding code in GetDist.f90).
  
  
• If your variable has a prior which cuts off when the posterior is non-negligible you need to set the limitsxxx variables to the corresponding limits - for example for the neutrino mass where m_v >0. Otherwise limits are computed automatically. DO NOT use limitsxx to change the bounds on other plots - the program uses the limits information when it is estimating the posteriors from the samples. When limitsxx are used the .margestats file contains one tail statistics. If only one end is fixed you can use N the floating end, e.g. "limitsxx = 0 N" for the tensor amplitude which has a greater than zero prior.

Output Text Files

• file_root.margestats file contains the means, standard deviations and marginalized limits for the different parameters (including derived parameters).
  
  
• file_root.likestats gives the best fit sample model, its likelihood, and limits from the extremal values of the N-dimensional distribution. Note that MCMC does not generally provide accurate values for the best-fit.
  
  
• file_root.converge contains various convergence diagnostics
  
  
• file_root.PCA contains a principle component analysis (PCA) forPCA_num parameters, specified in PCA_params parameter. You can specify the mapping used with PCA_func, for example 'L' to use logs (what you usually want for positive definite parameters). The output file contains the analysis including the correlation matrix, eigenvectors and eigenvalues, as well as the well constrained parameter combinations and their marginalized errors. This is how you automatically work out constraints of the form param1^a param2^b param3^c ...  = x \pm y.
  
  
• file_root.corr contains parameter correlations
  
  
• file_root.covmat contains a covariance matrix you can use as a proposal matrix for generating future chains

Plotting

• SuperMongo 1D plots
  GetDist produces a file_root.sm file for use with sm. Run sm < file_root.sm to produce file_root.ps containing a plot of the 1D marginalized posterior distributions.
  
  
• Matlab plots
  GetDist produces MatLab '.m' files to do 1D, 2D and 3D plots. Type file_root into a MatLab window set to the directory containing the .m files to produce 1D marginalized plots. You can also do
  • file_root_2D for 2D maringalized plots
  • file_root_3D to make coloured 2D samples plots (like the ones on the home page)
  • file_root_tri to make 'triangle' plots (example. Set triangle_plot=T in the .ini file)
  You can use the blue matlab script (in the mscripts) directory to change to a B&W-friendly colormap (see also other colormaps in that directory). To compare two different sets of chains set compare_num=1 in the .ini file, and compare1 to the root name of some chains you have previously run GetDist on.
  
  
• Custom Matlab plots
  Some matlab scripts are also supplied for making custom matlab plots using the files produced by GetDist (see also CosmoloGui). The scripts are in the mscripts directory - you will probably want to add this to your matlab path using e.g. addpath('mscripts'). The scripts are:
  • confid2D makes marginalized contour plots like this. For example confid2D('file_root',8,17,'-k','b') plots the 8th parameter against the 17th parameter (by default n_s against σ₈), using black line colour, and filling the contours in dark and light blue. Using the last parameter 'bg' would instead fill the contours in blue and green, or you can specify custom colors in RGB form like [.15 .15 .5; 0 1 1]. This script uses the contourf_col script supplied for making filled contour plots with specific colours. You can overlay contours from different data on top of each other, for example

    confid2D('file_root1',8,17,'-k','b');
    hold on;
    confid2D('file_root2',8,17,'-k','r');
    hold off;
    show_contours_behind;
    

    This last (optional) command is a supplied script which will show dotted lines lying behind other solid contours. If the last colour argument is omitted, confid2D plots unfilled contours only.
  • plot4D generates a rotatable 3D plot, coloured by a fourth parameter, like this. For example plot4D('file_root',16,20,17,4) would plot parameters 16, 20 and 17 on the x, y and z axes, coloured by parameter 4 (as in the example plot). Note that plot4D uses the file file_root_single.txt, which is generated by GetDist only when you explicitly thin the chains, or generate at least one 3D plot.
    

Parameter labels are set in distparams.ini - if any are blank the parameter is ignored. You can also specify which parameters to plot, or if parameters are not specified for the 2D plots or the colour of the 3D plots getdist automatically works out the most correlated variables and uses them. The data files used by SuperMongo and MatLab are output to the plot_data directory.

Convergence diagnostics

• For multiple chains the code computes the Gelman and Rubin "variance of chain means"/"mean of chain variances" R statistic for each parameter using the second half of each chain (reference). The .converge file produced by getdist contains the numbers for each parameter individually. The program also writes out the value for the worst eigenvalue of the covariance of the means, which should be a worst case (catching poor chain coverage in directions not aligned with the base parameters). This "R-1" statistic is also used for the stopping criterion when generating chains with MPI. If the numbers are much less than one then the second half of each chain probably provides an accurate estimate of the means and variances. If the distribution is bimodal and no chains find the second mode low values can be misleading. Typically you want the value to be less than 0.2. Low values do not guarantee that small percentiles will be measured accurately (though it certainly helps), or that the value won't increase as you run longer and probe further into the tails.
  
  
• For MPI runs, in addition to specifying a stopping criterion as above, you can also give a convergence criterion for a confidence limit. In general it is much harder to get an accurate value for a small confidence value than for the mean, so imposing a tight limit may make your chains run for a very long time (though you can then be pretty confident in your confidence). The value computed is the variance of the chain percentiles from the last half of the different chains, in units of the standard deviation of each parameter. You can either specify a particular parameter to check, or do all of them. Any parameter with a slowly explored tail will only converge very slowly, in which case you may be able to improve things by generating chains at a higher temperature (so the tails are explored more readily; GetDist can adjust for the temperature automatically), or re-parameterizing.
  
  
• For individual chains (before importance sampling) getdist computes the Raftery and Lewis convergence diagnostics. This uses a binary chain derived from each parameter depending on whether the parameter is above or below a given percentile of its distribution. The code works out how much the binary chain needs to be thinned to approximate a Markov process better than a second order process, and then uses analytical results for the convergence of binary Markov chains to assess the burn in period. It also assesses the thin factor needed for the binary chain to approximate an independence chain, which gives an idea of how much you need to thin the chain to obtain independent samples (i.e. how much you can get away with thinning it for importance sampling, though thinning isn't entirely lossless). The .converge file contains the numbers for each chain used, getdist writes out the worst case. See this reference for details.
  
  
• A simpler way to asses the error on quantile limits is by computing them using subsets of the samples. GetDist produces the file_root.converge contains the 'split-test' rms change on the parameter quantiles in units of the standard deviation when the samples are split into 2,3 or 4 parts. Small numbers are clearly better. A number of 0.5 for a 2-way split indicates that the 95% confidence limit may in fact be off by the order of half a standard deviation (though probably better, as the limit is computed using all the samples). Large values in a particular parameter may indicate that there is a long tail that is only being explored slowly (in which case generating the chain at a higher temperature or changing parameters to the log might help).
  
  
• The file_root.converge file also outputs the parameter auto-correlations as a function of step separation (see e.g. astro-ph/0310723). The correlation length is similar to the Raftery&Lewis statistics in giving a measure (though harder to quantify) of how far apart independent samples are, though it is less sensitive to how well the tails are explored (see bad case example in the notes).

Differences between GetDist and MPI run-time statistics

GetDist will cut out ignore_rows from the beginning of each chain, then compute the R statistic using the last half of the remaining samples. The MPI run-time statistic uses the last half of all of the samples. In addition, GetDist will use all the parameters, including derived parameters. If a derived parameter has poor convergence this may show up when running GetDist but not when running the chain (however the eigenvalues of covariance of means is computed using only base parameters). The run-time values also use thinned samples (by default every one in ten), whereas GetDist will use all of them. GetDist will allow you to use only subsets of the chains.

The above parameterization does make use of some knowledge about the physics, in particular the (approximate) formula for the sound horizon. Also supplied is a params_H.f90 file which uses H_0,z_re and A_s instead of theta, tau and log(10^10 A_s) which is more generic. Though slower to converge, this may be useful if you want to play around with different extended models - just edit the Makefile to use params_H.f90 instead of params_CMB.f90. Sample input files and covariance matrix along with params_H.f90 are available here. Since the parameters have a different meaning in this parameterization, you should not try to mix .covmat (or other) files with those from the default parameterization.

Hard coded priors

The default installation hard codes a few priors, in some instances you may wish to edit these:

• 40 < H_0 < 100 This is hard coded in params_CMB.f90 (ParamsToCMBParams routine for solving for H from theta), and also in calclike.f90.
• -0.3 < Omega_k < 0.3 in calclike.f90
• 10Gyr< age < 20Gyr in calclike.f90, but only used if you've set Use_Age_Tophat_Prio=T in the .ini file
• chi < 3.14 In a closed universe, the code does not allow models where the photons wrap around

There is no prior on the positivity of Omega_Lambda.

The supplied CMB datasets that are used for computing the likelihood are given in *.dataset files in the data directory (these may not be up to date). These are in a standard .ini format, and contain the data points and errors, data name, calibration and beam uncertainties, and window file directory. Code for handling these is in cmbdata.f90. The WMAP data is handled separately as a special case. Various simple priors are encoded in calclike.f90.

There is also built-in support for 2dF and (few years old) supernovae observations. Adding new data sets should be quite straightforward - you are encouraged to donate anything you add to be used by everyone.

The most likely need to modify the code is to change l_max, num_cls, or matter_power_lnzsteps, all specified in cmbtypes.f90. To change the numbers of parameters you'll need to change the constants in settings.f90. Run "make clean" after changing settings before re-compiling. When adding just one additional parameter it's often easiest to re-interpret one of the default parameters rather than adding in new parameters.

You are encouraged to examine what the code is doing and consider carefully changes you may wish to make. For example, the results can depend on the parameterization. You may also want to use different CAMB modules, e.g. slow-roll parameters for inflation, or use a fast approximator. The main source code files you may want to modify are

• params_CMB.f90

This defines what the input variables mean. Change this to use different variables. You can change which parameterization file to use in the Makefile. params_H.f90 is also supplied for using z_re, A_s and H_0 instead of tau, log(A_s) and theta.
• cmbtypes.f90

You need to change this file to specify the l_max used. Chains can be generated at low l_max, then post-processed with a compile using a higher l_max. You can also change the num_cls number of (temperature plus polarization) Cls to compute and store.
• settings.f90

This defines the number of parameters and their types. You will need to change this if you use more parameters.
• cmbdata.f90

This reads in the CMB .dataset information and computes likelihoods. You may wish to edit this, for example to use likelihood distributions for the band powers, or to compute the likelihood from actual polarized data. This version assumes polarized data points are an arbitrary combination of the raw TT, TE, EE, and BB Cls, as specified in the window files in data/windows. WMAP data is handled as a special case.
• mpk.f90

Analagous to cmbdata, but for matter power spectrum measurements. Reads in generic dataset files, supported (fixed) covariance matrices.
• propose.f90

This is the proposal density and related constants and subroutines. The efficiency of MCMC is quite dependent on the proposal. Fast+slow and fast parameter subspaces are proposed separately. See the notes for a discussion of the proposal density and use of fast and slow parameters.
• CMB_Cls_simple.f90

Routines for generating Cls, matter power spectra and sigma8 from CAMB. Replace this file to use other generators, e.g. a fast approximator like CMBfit or DASH.
• supernovae.f90

As of May 2006, uses SNLS by default. Edit to use Riess gold. (thanks to David Rapetti, Jochen Weller and Anze Slosar).
• lya.f90

Lyman alpha data (thanks to Matteo Viel, J.Lesgourgues). Can also replace with SDSSLy-a-v3.f90 and recompile to use SDSS data (thanks to Kevork Abazajian). Note this is only tested and likely to be reliable for standard LCDM models.
• postprocess.f90

Reads in .data files and re-calculates likelihoods or theory predictions. Unused in MCMC runs.
• calclike.f90

Add in calls to other likelihood calculators, etc., here.
• driver.F90

Main program that reads in parameters and calls MCMC or post-processing.
• GetDist.f90

The "getdist" program for analysing chains. Write your own importance weighting function or parameter mapping.

• May 2006 Supernovae.f90 updated to use SNLS by default, edit to use Riess Gold. 2dF updated (twodf.f90 file deleted, use 2df_2005.dataset); covariance matrix support in mpk.f90. Fixed bug using LSS with non-flat models. Improved error checking and matlab 7 enhancements in getdist. Getdist auto column counting with columnnum=0, various hidden extra options now shown in sample distparams.ini. Extra fix for confid2D. Fixed MPI thinning bug in utils.F90. Makefile fixes. Fixed mpk.f90 analytical marginalization (since March 2006). SDSS likelihood now computed from k/h=1e-4.
• April 2006 Fixed bug in lya.f90 (SDSS lyman-alpha now the default; lya.f90 now includes Croft by default). Fixes to Confid2D matlab script. Added .covmat files for WMAP with running and tensors, and basic Planck simulation. Fixed version confusion in GetDist (one-tail limits set to prior limit value).
• March 2006 Updated for 3-year WMAP. Added use_lya to include lyman-alpha data (standard LCDM only). Default in lya.f90 is LUQAS (can also compile with SDSSLy-a-v3.f90 for SDSS). New matlab scripts for producing solid contour and 4D plots. New checkpoint option to generate checkpoint files and continue terminated runs. Added action=1 parameters redo_add (adds new likelihoods rather than recalculating) and redo_from_text (if you don't have .data files). Added pivot_k and inflation_consistency for use with default power spectrum parameterization.
• July 2005 Added get_sigma8 to force calculation of σ₈. Updated .newdat CMB dataset format (also added B03 data files). New use_fast_slow parameter to turn on/off fast-slow optimizations. Fixed bug which resulted in occasional wrong tau values when importance sampling .data files. GetDist now outputs one/two-tail limit info in .margestats file. Updated CAMB version (support for non-linear matter power spectrum).
• February 2005 Updated CAMB for new accurate lensed C_l calculation of astro-ph/0502425. Minor changes to getdist (new matlab_version input parameter, all_limits to set same limits for all parameters). cmbdata.f90 includes new format used by BOOMERANG/CBI for polarization.
• October 13 2004 Fixed bug in mpk.f90 when using 2df. Changes to GetDist for compatibility with MatLab 7. Fixed Makefile_intel (though now obsolete if you have Intel fortran v8).
• October 2004 Added mpk.f90 for reading in general (multiple) matter power spectrum data files in a similar way to CMB dataset files - corresponding changes to input parameter file. Included SDSS data files (note CosmoMC only models linear spectrum). Various minor bug fixes and improved MPI error handling. Included (though not compiled by default) supernovae_riess_gold.f90 file to include more recent supernova data. Some mscripts fixes for compatibility with MatLab 7.
• August 2004 Improved proposal density for efficient handling of fast and slow parameters, plus more robust distance proposal (should see significant speed improvement). New sampling_method parameter: new options for slice sampling (robust) and directional gridding (robust and efficient use of fast parameters). Also option to use slice sampling for burn in (more robust than Metropolis in many cases), then switch to Metropolis (faster with good covariance matrix). See the notes for details. Improved MPI handling and minor bug fixes. Fixed effect of reionization on CAMB's lensed C_l results.
• June 2004 Uses June 2004 CAMB version: bessel_cache.camb file no longer produced or needed (prevents MPI problems). Increased sig figs of chain output and GetDist analysis. New parameter propose_scale, the ratio of proposal width to st. dev., default 2.4 (following Roberts, Gelman, Gilks, astro-ph/0405462) - often significantly speeds convergence (parameters in .ini file are now estimates of the st. dev., not desired proposal widths). Added MPI_R_StopProposeUpdate to stop updating proposal covariance matrix after a given convergence level has been reached. Added accuracy_level parameter to run at higher CAMB accuracy level (may be useful for forecasting).
• March 2004 Added new VSA and CBI datasets. Added first_band= option to .dataset files to cut out low l that aren't wanted. CAMB pivot scale for tensors changed to 0.05/MPc (same as scalar). Fixed various compiler compatibility issues. Corrected CMB_lensing parameter in sample .ini file. Fixed minor typo in params_CMB.f90. Fixed reading in of MPI_Limit_Converge parameter in driver.F90. Fixed bounds checking in MatterPowerAt (harmless with 2df). Added an exact likelihood calculation/data format to cmbdata.f90 for polarized full sky CMB C_l.
• December 2003 Added MPI support, with stopping on convergence and optional proposal density updating. Added calculation of matter power spectrum at different redshifts using CAMB (settings in cmbtypes.f90). Fixed bug when restarting chains using "continue_from" parameter [March 2006: now obsolete], and a few compiler compatibility issues. Updated CAMB for more accurate non-flat model results. Added output of parameter auto-correlations to GetDist, along with support for ignore_rows<1 to cut out a fraction of the chain and percentile split-test error estimators. Changed proposal density to proposal a random number of parameter changes on each step. Added GetDist samples_are_chains option - if false, rows can be any samples of anything (starting in column one, without an importance weight or likelihood value as produced by CosmoMC) - useful for analysing samples that don't come from CosmoMC. Added GetDist auto_label parameter to label parameters automatically by their parameter number.
• July 2003 Fixed bug in MCMC.f90 affecting all raw chains - weights and likelihoods were displaced by one row. Post-processed results were correct, and effect on parameters is very small. Minor bug fixes in GetDist. Can now make file_root.read file to be read by all chains file_root_1, file_root_2, etc (this file is not auto-deleted after being read).
• May 2003 Added support for 'triangle' plots to GetDist (example. Set triangle_plot=T in the .ini file). If truncation is required, the covariance matrix for CMB data sets is now truncated (rather than truncating the inverse covariance). Fixed CAMB bug with non-flat models, and problem setting CAMB parameters when run separately from CosmoMC.
• March 4 2003 Fixed bug in GetDist - the .margestats file produced contained incorrect limits (the mean and stddev were OK)
• Feb 2003 Support for WMAP data (customized code fixes TE and amplitude bugs). CMB computation now uses C_l transfer functions - complete split possible between transfer functions and the initial power spectrum, so improved efficiency handling fast parameters. Bug fixes and tidying of proposal function. Initial power spectrum no longer assumed smooth for P_k. GetDist limitsxxx variables can be N to auto-size one end (margestats are still one tail). Support of IBM XL fortran (workarounds for bug on Seaborg). GetDist will automatically compute some chain statistics to help diagnose convergence and accuracy. CAMB updated, including more accurate and faster handling of tight coupling. Option to generate chains including CMB lensing effect. Various other changes.
• Nov 2002 Added support for polarization, and improved compatibility with different compilers and systems.

Information Theory, Inference and Learning Algorithms

MCMC Preprint Service

Raftery and Lewis convergence diagnostics

There are also some notes on the proposal density, fast and slow parameters, and slice sampling as used by CosmoMC. See also the BibTex file of CosmoMC references you should cite, along with some references of potential interest.

FAQ

1. What are the dotted lines on the plots?
   Dotted lines are mean likelihoods of samples, solid lines are marginalized probabilities. For Gaussian distributions they should be the same. For skew disctributions, or if chains are poorly converged, they will not be. Sometime there is a much better fit (giving high mean likelihood) in a small region of parameter space which has very low marginalized probability. There is more discussion in the original paper.
    
2. What's in the .likestats file produced by getdist?
   These are values for the best-fit sample, and projections of the n-Dimensional confidence region. Usually people only look at the best-fit values. The n-D limits give you some idea of the range of the posterior, and are much more conservative than the marginalized limits.
    
3. I'm writing a paper "constraints on X". How do I add X to cosmomc?
   Often the easiest thing to do is to re-interpret one of the unused standard parameters, e.g. w, A_T, etc, depending on whether the parameter is "fast" or not (if in doubt, use one of the slow parameters like w or f_nu). You just need to change the CMBToCAMB routine in CMB_Cls_Simple.f90 so that your parameter is correctly fed into CAMB, change limits appropriately in the params.ini file, etc. See the undocumented references to w_is_w in the code for how w can be re-interpreted as a ratio of isocurvature to adiabatic in CAMB's initial conditions. If you need to add more than a couple of parameters you'll probably instead need to edit settings.f90 to increase the number of parameters, and edit the .ini file accordingly. Slow parameters should have numbers lower than fast parameters (i.e. to insert 5 slow parameters, the index of the fast parameters in the .ini file should increase by 5).
    
4. How do I submit cosmomc jobs to the LSF Cambridge COSMOS supercomputer queues?
   The data files are already on cosmos, so just download the code from LAMBDA and then use 'ln -s /home/cosmos-tmp/WMAP data' to make the link from the WMAP/data directory to the existing files. To run use the runCosmomc script that should be installed on your path: type just 'runCosmomc' for command options. Use 'module load cosmolib' and linking options are

   cfitsio = /home/cosmos/share-ia64
   healpix = /home/cosmos/share-ia64/Healpix_2.01
   WMAP3 = /home/cosmos/ccc-cam/aml1005/WMAP3
   

5. Why do some chains sometimes appear to get stuck?
   Usually this is because the starting position for the chain is a long way from the best fit region. Since the marginal distributions of e.q. A_s are rather narrow, it can take a while for chains to move from into an acceptable region of A_s exp(-2τ). The cure is to check your starting parameter values and start widths (make sure the widths are not too wide), or to use a sampling method that is more robust (e.g. use sampling_method = 4). If you are patient, stuck chains should eventually find a sensible region of parameter space anyway. Occasionally the staring position may be in a corner of parameter space so that prior ranges prevent any resonable proposed moves. In this case check your starting values and ranges, or just try re-starting the chain (a different random starting position will possibly be OK).
    
6. How to I simulate futuristic CMB data? See CosmoCoffee.

Feel free to ask questions (and read answers to other people's) on the CosmoCoffee software forum. There is also a FAQ in the CosmoloGUI readme.