ANNEX A: METHODOLOGY
This report has been made possible by reporting from Australia’s development NGOs. It has also been made possible by the data collation efforts of ACFID, the ACNC, DFAT and the Australian National University’s Development Policy Centre.
This report draws upon datasets compiled by these organisations. Each of the four central datasets is outlined below, alongside data issues. We also briefly explain the statistical measures used in the
report. Footnotes are used throughout the report to explain any additional specific data issues. Where a dataset is only used once in the report (this is true of DFAT data and some public opinion surveys) the dataset is detailed in the section of the report where its findings are discussed.
The ACNC dataset
All Australian charities and NFP organisations are required to provide an AIS to the ACNC each year. Approximately, 55,000 organisations provide this information to the ACNC; the end result is an impressive dataset. However, the resulting data presented some challenges for our use. The dataset contains comparatively few variables that speak to the indicators chosen for this report. Also, it does not have a single variable that allows development NGOs to be distinguished from other types of NFP organisations. It is relatively easy to distinguish development NGOs from organisations that work solely in Australia. It is harder to distinguish development NGOs from other organisations that work overseas, but which focus most of their work on wealthy countries. Isolating organisations that primarily undertake religious work is also difficult. We isolated development NGOs using a complex combination of filters. The end result, we believe, is a good approximation, although it is imperfect. (Our numbers are slightly higher than those of other researchers such as Knight and Gilchrist (2015, p. 3) and Cortis et al. (2015, p. 9). However, our estimates are of the same order of magnitude.) We have drawn on ACNC data when discussing the number of NGOs, staffing, and volunteer numbers. Owing to the challenges of isolating development NGOs, we have only used one year’s ACNC data in our reporting.
The ACFID Statistical Survey
The ACFID Statistical Survey is a detailed survey that ACFID members complete every year. It contains questions about many aspects of development NGO work, including questions about overall
organisation attributes as well as individual projects. In recent years, ACFID staff have carefully validated data gathered in this survey. As a result of the detailed data it gathers, the Statistical Survey forms the backbone of this report. Although Statistical Surveys have been run for many years, compatibility issues mean that we have only been able to use Statistical Survey data since 2013. The most recent Statistical Survey data is for 2016. Occasionally, ACFID members have failed to provide information for a specific year or have failed to provide information for all years. We have taken precautions, outlined below, to ensure that missing data has not biased our findings.
The Electorate Snapshot survey
In 2016, ACFID surveyed 19 of its largest members asking them questions about the number of public donors they had in each electorate in 2015. They also asked about support from schools, and church and community groups. This information provides important insights into variation in support between different parts of Australia. In the report we focus on variation between states.
One limitation of the Electorate Snapshot data is that individuals and entities that engaged with more than one of the surveyed NGOs in 2015 may have been double counted. Another limitation is that Electorate Snapshot Data only comes from a subset of Australia’s development NGOs and as a consequence probably under-represent engagement to a degree. However, because the organisations sampled are amongst the largest development NGOs, the subset is large. In terms of donations raised, it is estimated that over 80% of ACFID member donations go to the surveyed organisations, and that over 60% of all development NGO donations are raised by the surveyed NGOs (Wood et al. 2016b, p. 10). As a result, while the Snapshot Survey data under-represents total community engagement to a degree, it still provides a good starting point for understanding levels of engagement. Moreover, to some extent, the over-representation and under-representation outlined above will offset each other. Most importantly, the sample of NGOs is also large enough that the data ought to provide reasonable estimates of relative levels in engagements between different states.
ACFID Code of Conduct reporting
Every year all ACFID members are required to report on their compliance with the ACFID Code of Conduct. ACFID aggregates individual members’ responses into a reporting database. In some
instances, we were able use aggregate information from these responses to report against indicators. Data from the Code of Conduct covers all ACFID members. Because of changes to the Code of Conduct in 2017, the most recent data we had available this year came from Code of Conduct reporting in 2015. Future State of the Sector reports will be able to draw on more recent data.
The Development Policy Centre dataset
The Development Policy Centre has created a dataset–based on ACFID data, and on data fromdevelopment NGOs’ annual reports that track total revenue and total public donations over time. The
advantage of this dataset is that it is a long time seriesstemming back to 2002. In addition to including all ACFID members in its calculations of totals, it also includes data on the two largest development NGOs that are not-members of ACFID, making it more representative of the sector as a whole. However, the dataset only contains information on public donations and total revenue, and only provides aggregate totals, as opposed to information on individual NGOs. This means that it is only useful in studying overall revenue trends.
Understanding the calculations
Totals, means and medians
The Australian NGO sector is comprised of a small number of very large organisations as well as many smaller organisations. Often, we break our charts down by different size categories. (See the
boxed text on ACFID size categories.) However, at times there is also considerable variation within the size categories. Because of this, to ensure the most representative possible picture, we use different statistics in different parts of the report depending on the information available to us and the nature of the data.
- When we’re interested in the total sector (or totals for each size group) we report on aggregate totals.
- When we’re trying to get a sense of the state of affairs in the average development NGO we report on means (the most common measure of average) whenever we can. At times though, unusual organisations bias the calculations of means in a way that makes means unrepresentative. When this problem occurs, we use medians (the median is the middle organisation in a group once organisations have been sorted on the variable of interest).
- At times, the traits of different development NGOs vary a lot. When seeing that this variation is important, we have charted it using histograms.
Trends over times
Where possible, we report on trends over time. Sometimes this is not possible because we only have one year’s data. In other instances, because we often only have data for four years, trends – if they exist – are not sufficient to be distinguished from random fluctuations. When we only have one year’s data, or when trends aren’t obvious, we often only report on the most recent year.
When we report on trends using ACFID Statistical Survey data, we only include organisations that have provided data for every relevant year. This is to ensure that apparent trends are not artefacts of changes in the organisations providing data. Also, when we report on trends for different size groups, we keep individual organisations in the same size groups over time. This is to ensure that apparent trends are not simply a product of changes in group composition.
When we report on trends relating to financial data, we provide inflation-adjusted figures to ensure that we are capturing real change.