Assessing Community Model Goodness-of-Fit

Goodness-of-fit (GoF) tests help determine how well your fitted community occupancy model represents the observed detection patterns in your data. This involves comparing your actual data to new datasets simulated from the model's estimated parameters.

Step-by-Step Guide to GoF Testing

  1. Configure Settings
    • Number of posterior draws: Select how many samples from the model's posterior distribution to use for simulating new datasets. More draws increase precision but take longer. 1000 is often a reasonable starting point.
    • Condition on occupancy (z):
      • If TRUE, the simulation conditions on the estimated occupancy state (z) for each species at each site. Tests only the detection component of the model, fixing occupancy/abundance to estimates from the model, rather than generating them anew
      • If FALSE, the test "generates new z for each posterior sample (testing complete model, including detection and occupancy submodels)." This simulates new occupancy states based on your occupancy submodel parameters before simulating detection data, providing a more comprehensive test of the entire model structure.
      • A two-step approach can be informative: first test with conditioning (TRUE) to isolate detection fit issues, then test without conditioning (FALSE) to evaluate the complete model if detection fit is adequate.
    • Residual type: Choose the metric for comparing observed and predicted data. Freeman-Tukey ("FT") residuals are often recommended. Pearson Chi-squared residuals are offered as an alternative.
  2. Run the Test
    • Click "Run Goodness of Fit Test" to perform the calculations. This can take some time, especially with many draws or species.
  3. Interpret Results
    • Overall Model Fit (Community p-value): This value indicates the overall fit across all species. It's the proportion of simulations where the discrepancy (sum of residuals) was greater than or equal to the discrepancy in your observed data.
      • Values near 0.5 suggest a good fit.
      • Values < 0.1 or > 0.9 indicate potential lack of fit (model doesn't capture data patterns well).
      • Values < 0.1 might suggest the model predicts more variation than observed (underdispersion).
      • Values > 0.9 might suggest the model predicts less variation than observed (overdispersion).
    • Species-level Results: This table shows the Bayesian p-value for each individual species, helping identify which species might be poorly fit even if the overall fit seems adequate. Interpretation follows the same principles as the overall p-value. The table also includes an interpretation category ("Excellent", "Good", "Moderate", "Lack of fit") and dispersion pattern indication.
    • Residual Plots: These plots provide a visual check of model fit for each species by comparing observed vs. predicted residuals.
      • Each point typically represents the average residual (discrepancy) for a specific site.
      • The X-axis shows the residual calculated from your actual observed data.
      • The Y-axis shows the residual calculated from new data simulated by the fitted model.
      • The dashed red line (y=x) indicates perfect agreement.
      • Good Fit: Points should scatter closely and randomly around the red line. This means the model simulates data with similar discrepancy patterns as your real data.
      • Poor Fit (Predicted > Observed): Points consistently fall above the red line. This suggests the model predicts more variation/structure than observed (potentially related to underdispersion in data or model misspecification) and often corresponds to a low Bayesian p-value (< 0.1).
      • Poor Fit (Observed > Predicted): Points consistently fall below the red line. This suggests the model fails to capture variation/structure present in the observed data (potentially related to overdispersion in data or model misspecification) and often corresponds to a high Bayesian p-value (> 0.9).
      • Wide Scatter: Points are widely spread far from the red line, indicating inconsistent model performance.
      These plots help visualize why a species might show lack of fit based on its p-value.

Important Notes:

Tips if Fit is Poor: