Framework
The study adopts an ecological theoretical framework [13], which asserts that a child’s developmental attainments and well-being are embedded within the contexts of the family, the ECEC program and the broader social and economic community. A key feature of the design is that it positions the evaluation of ECEC programs within diverse communities, across Victoria and Queensland, selected on the basis of both advantage and their “risk” to children’s outcomes [14] (population characteristics) and program access (location). Children not attending ECEC programs were selected as a no-program control [NPC], and their care environments and outcomes were measured. Concurrent economic data will enable accurate analyses of on-going investment effectiveness supported by longitudinal cost-benefit analyses.
Sampling methodology
To address the key research aims of the E4Kids study, a cluster-randomised sampling design was used to select a cohort of children attending typical or ‘everyday’ ECEC programs. The cohort was recruited in 2010 and participated longitudinally until child-records data linkage in 2015. The process used to achieve the final sample involved identifying: (1) the target population (for which the results from E4Kids intend to be generalised), (2) the sampling frame that represents that target population (the achieved population), (3) the target sample, and (4) the achieved sample. This approach to sampling is based on other large education studies [15, 16].
E4Kids focuses specifically on children participating in approvedFootnote 1 ECEC programs in Australia. Therefore, the target population included a subset of children participating in ECEC programs in Australia. This is an important distinction for E4Kids since other large Australian data sets include more general information about children and their development. The scope of the target population was reduced by three contextual factors: population density constraints, child age constraints and funding and access constraints. Population density constraints reduced the scope of the target population because very remote areas of Australia have low population densities and no access to everyday ECEC provision. Some very remote areas receive other forms of provision, including mobile or visiting services, while others receive no provision at all [17]. Areas that did not provide typical everyday ECEC programs were excluded from the target population.
Child age constraints reduced the scope because of the variability of the ages of children participating in different forms of ECEC provision. To normalise the age ranges of children from different provision types and ensure all major provision types were included in the study, the target population was reduced in scope to include children who participated in ECEC classrooms that usually included 3–4 year old children. By implication, this excluded, for example, infant classrooms in long day care services.
Funding and access constraints reduced the scope of the target population by limiting the total size of the study. Yet to maintain the integrity of estimates and achieve generalisable results, a sufficient number of classroom-child observations needed to be made. Since the study was part-funded by the State Government jurisdictions of Queensland and Victoria, the target population was limited to within these states. To maximise the available budget, minimise the need for travel between sites, and to produce a sample that was representative of the diversity within Australia, regions were deliberately selected as the study’s sites including the Statistical Divisions of major metropolitan Queensland and Victoria (metropolitan); and the Statistical Local Areas of a greater regional area in Victoria (regional) and a remote location in Queensland (remote).
The achieved population was sourced from regulatory lists of licensed ECEC programs in the four study regions. These lists – provided by the State Government partners and current for the year 2009 – comprised the sampling frame. The sampling frame was explicitly stratified by location (metropolitan, regional, and remote) and service type (LDC, K, FDC, ODC). Some minor forms of ECEC services were excluded (Early Childhood Inclusion Services and Restricted Licenses in Victoria, representing less than 1% of all programs) as per the scope of the sample design. This yielded 16 explicit strata.
A target sample of 150 services and 2,500 children was set based on the likely range of the design effect, to ensure that sample estimates would be generalisable. This target sample was split proportionally between each stratum to establish the target numbers of ECEC services within each stratum presented in Table1. Within each explicit stratum, implicit stratification was used to ensure a spread of services from high and low SES (Socio-Economic Status) neighbourhoods. Each stratum was weighted by neighbourhood SES and service capacity, to ensure that ECEC services in the highest and lowest quartiles of SES would be included in the sampling process.
Stage one of sampling occurred from September to December 2009 and involved the random selection of ECEC services of proportional size (as measured by the total licensed capacity in each stratum) from the sampling frame. Within the sampling frame, with services now listed and weighted by neighbourhood area SES, a new vector was created to represent the weighted cumulative sum of the capacity in each stratum and ranged from one to the sum of the weighted capacity for each stratum. A random number within the range of the cumulative sum vector was drawn to comprise the first service sampled. The remaining target number of services was sampled by going down the list of ECEC provider names using a pre-determined sampling interval, and looping back into the top of the sampling frame when the bottom of the list was reached.
Information letters were sent, and follow-up phone calls were employed to each selected ECEC service provider to explain the study and to invite the director of the ECEC service to participate. Services that did not agree to participate where replaced by the service that was listed next on the sampling frame. If the replacement service did not agree to participate either, then the next replacement was the service listed above the originally sampled service in the sampling frame. This ‘nearest neighbour’ replacement strategy was used until a service that was similar to the first sampled service agreed to participate. Table1 shows that a minimal total replacement sampling was conducted in the study. However, when replacement sampling was required, it was usually the next service listed in the sampling frame that agreed to participate.
Stage two of sampling was conducted in the first quarter of 2010. It involved recruiting clusters of children, aged three and four, from classrooms in the services (that agreed to participate) in stage one. Each of these services was audited using a standardised schedule that listed all possible characteristics of an ECEC classroom – for example, the type, capacity, and an age-range of all classrooms in each service were recorded. Classrooms that included five or more children between the ages of three and four were included in E4Kids and all children in selected classrooms were invited to participate. In FDC situations, households were recruited if they included as least one child aged between three and four.
This process of sampling achieved a sample of 2,494 children, drawn from 142 recruited services, for E4Kids. The longitudinal nature of the design meant that the services and classrooms included in subsequent years were non-randomly selected; as participating children progressed into preschool and school classrooms selected by their families, these services and classrooms were consequentially recruited into the study. In 2011 and 2012, 721 and 806 ECEC and schools services, respectively, participated. Within these ECEC and school programs, there were a total of 286; 1,136; and 1,427 classrooms in 2010, 2011, and 2012, respectively. The study continued in 2013, 2014 and 2015, including data linkage with school sector evidence on children’s progress and performance. A summary of the achieved sample is given in Tables2 and 3.
The achieved sample was split approximately equally by gender: 1,199 females (48%), 1,294 males (52%) and 1 non-response. Children’s ages at January 1 in each year of the study are shown in Table4, and reinforces the diversity in ages when recruiting children participating in everyday ECEC programs in Australia that include children aged three to four.
Sampling of the No Program Control (NPC) group
Children who did not attend a program were the NPC group. The best approximation for children not in approved care was the residual of a list of families receiving subsidy for approved care subtracted from a list of all families known to have children of a given target age. No single department held both pieces of data: the Department of Education, Employment and Workplace Relations (DEEWR) held records of the families who received subsidy for care in the Child Care Management System (CCMS) and the Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA) held records of families with children of given ages in the Family Tax Benefit administrative records. From the residual group, it was necessary to subtract those families who used kindergarten programs (funded by state government) and those who did not use subsidies but used approved care. Consultation with FaHCSIA and DEEWR suggested that the recruitment rate of the target sample could be as low as 3%. Therefore, it was decided to deliberately oversample to offset an overly low recruitment rate and ensure a reasonable achieved sample size.
The NPC sample was explicitly stratified by location and age to mirror the E4Kids sample. Nine-hundred families were selected and stratified: 346 each from the greater urban area in each state, and 104 each from regional Victoria and remote Queensland. In addition, the children from families needed to fit within the following age ranges, in recognition of the entry conventions in each State: in Queensland, children should be born after June 30, 2006 and before June 29, 2007; in Victoria, children should be born after April 30, 2007 and before April 29, 2008.
Nine-hundred families represented 4.32% of the sampling frame (N = 20,826), including 7.4% of the urban Queensland sampling frame (N = 4,661), 2.2% of the urban Victorian sampling frame (N = 15,668), 29.4% of the regional Victorian sampling frame (N = 353) and 72.2% of the remote Queensland sampling frame (N = 144).
After a two-week opt-out period, a two-staged recruitment process was undertaken. Initially, all sampled families were sent an E4Kids recruitment pack that included a statement about the study, a consent form and a short survey asking about their use of ECEC services to screen out any families whose children had previously participated in approved programs. Following this, all families who had not returned a consent form and screener, were phoned. When contact could not be made, a message was left (where possible) and families were followed up a maximum of three times (at different times or days, unless instructed otherwise) over a 2-week period. A final mail-out was conducted to all remaining families that had not been reached.
A screening tool was used to identify families who utilised kindergarten programs or approved programs but did not receive government subsidy; however, families remained eligible if they used any amount of informal care, including playgroups. Families were screened out if:
- 1.
They used approved care or kindergarten for more than 10h per week in a typical week, unless they had used these programs for less than 3months in 2010.
- 2.
The child fell outside the nominated age range.
Of the 900 NPC families sampled, 59 opted out via the FaHCSIA phone line. The greatest barrier to recruitment was contacting families: 364 (43%) of the 841 families sampled were unable to be reached. Of the remaining 477 families, 322 (67.5%) either declined to participate in the study, were screened out because of ECEC use or age ineligibility or did not return a consent form. One-hundred fifty-five families (32.5%)Footnote 2 agreed to participate and were recruited to comprise the NPC.
Weighting methodology
The methodology used in E4Kids to calculate sampling weights reflects the best standards of practice and aligns with international studies of educational achievement [15, 16]. Sampling weights were calculated for all children and services recruited in 2010. Services in subsequent years were recruited non-randomly (i.e. as a consequence of children moving into and through services, as mentioned previously). Therefore, in the cross-sectional years (2011 and 2012) services were equally weighted.
The service weight was interpreted as the number of services that each sampled service represented in the population. The weight of a service (i) was denoted, W i . For remote Queensland all services were selected with certainty and therefore W i equaled onef. In all other strata, the weight of a service was calculated as the product of a base weight, a correction factor and a trimming factor, as shown in Eq. 1. Where a service was selected and replaced by another service, the participating service inherited the weight of the originally sampled service.
Equation 1: Service weight function.
$$ {\boldsymbol{W}}_{\boldsymbol{i}}={\boldsymbol{w}}_{\boldsymbol{i}}{\boldsymbol{f}}_{\boldsymbol{i}}{\boldsymbol{t}}_{\mathbf{1}\boldsymbol{i}} $$
(1)
Where w i is the base weight of a service i that (approximately) sums across selected services in a stratum, to give the total number of services in the stratum, and is given by Eq. 2 below.
Equation 2: Service base weight function.
$$ {w}_i=\frac{int\left(\frac{{\displaystyle \sum }mos}{n}\right)}{mo{s}_{i\ }} where\ int\left(\frac{{\displaystyle \sum }mos}{n}\right)>mo{s}_{i\ }\ else\ 1 $$
(2)
\( int\left(\frac{{\displaystyle \sum }mos}{n}\right) \) is the sampling interval within the explicit stratum, given by the sum (within the stratum) of measures of size (the capacity of each service), divided by the number of services within the stratum.
One service, in regional Victoria, had mos i greater than the sampling interval, and received a base weight of 1 as per the conditional statement in Eq. 2. Thus the sum of selected service base weights was an approximation of the count of services within the stratum, with some random perturbations due to chance (e.g. start value and sampling intervals < mos i ).
f i was a correction factor to account for implicit oversampling of services in high and low SES communities. During sampling, services were ordered by the SES of the community they were a part of, as measured by the Socio-Economic Index for Areas (SEIFA) Index of Relative Socio-Economic Advantage and Disadvantage (IRSAD), and were then randomly selected proportionally to size (random start, selecting every service that includes the jth student – the sampling interval). The first and fourth quartiles, or IRSAD, within each stratum, were weighted greater than the middle quartiles in the proportions 35, 15, 15, 35. The correction factor was therefore 0.25/0.35 for services in the first and fourth quartiles, and 0.25/0.15 for services in the middle quartiles.
t1i was a trimming factor to reduce the weights of services with very large values of w i . Large values of w i occurred when services with very small mos i relative to other services in the stratum, were selected; they received very large base weights because of their low probability of selection. To compensate for this, the mean value for mos, M(mos), within the stratum was calculated, and services with mos i ≤ M(mos)/1.5 inherited a trimming factor equal to less than one, which reduced their influence on parameter estimates. Services with mos i > M(mos)/1.5 inherited a trimming factor equal to one. The trimming factor for services with a capacity less than 1.5 of the M(mos) were given by the ratio of the w i ’(the service base weight), with M(mos) replacing mos i . Therefore, the trimming factor can never be greater than one. Fifteen per cent of services in the sampling frame received a value for the trimming factor not equal to one. The formula for t1i is further explained by Eqs. 3 and 4.
Equation 3: Calculation of service weight prime for services with small measure of size.
$$ {w}_i\hbox{'}=\frac{int\left(\frac{{\displaystyle \sum }mos}{n}\right)}{\mathrm{M}(mos)} where\ mo{s}_i\ \le M(mos)/1.5\ else\ {w}_i $$
(3)
Equation 4: Function for service trimming factor.
$$ {t}_{1i} = \frac{w_i\hbox{'}}{w_i} $$
(4)
When calculated, the mean service weight in the achieved sample was 16.28 (SD = 15.59, min = 0.77, max = 72.62). Weights had a minimal impact on parameter estimates. Note that weighted estimates were given by non-parametric empirical bootstrap using 500 replications in the boot library of R [18].
Measures
Table5 presents a detailed summary of the measures and items selected for E4Kids, with corresponding explanations for the variables. Participating children were tested at least three times on standard outcome measures. Direct measures of the children included:
-
Cognition and achievement of individual children: Woodco*ck Johnson III (WJ-III; established standardised assessment tool) used each year, commencing April 2010.
-
Measurement of height, weight and waist circumference: recorded at each wave of data collection to identify children’s physical growth.
-
National numeracy and literacy scores (NAPLAN) in Year 3: obtained by data linkage from State Government partners.
In addition, the interaction amongst children (social inclusion and friendship) was measured using ‘Bus Story’ (a participation exercise; 2010 and 2011). However, the Bus Story tool was not used for the NPC. A parent survey was delivered to parents of participating children to gather family-related information. Adult-child interaction measures included:
-
Observational assessments of the quality of adult-child interactions in ECEC: Classroom Assessment Scoring System (CLASS).
-
Observation of adult-child interactions based on picture-story telling in ECEC: Thorpe Interaction Measure (TIM), applied in 2010 and 2011.
ECEC services information included:
-
Space and furnishings, personal care routines and activities in ECEC: Early Childhood Environmental Rating Scale – Revised.
-
Teacher/Educator survey (for educators working directly with the children).
-
Program Director or School Principal survey (included specific questions relating to the cost of approved care for the purpose of economic analysis).
-
Audit of the attendance of children in the programs.
Researcher training
More than 40 research assistants were employed each year to undertake data collection. Training on WJ-III, Bus Story and TIM was conducted over two full days. It comprised group training at the university and implementation piloting of each measure on children in LDC centres. The researchers assessed the children and submitted scored test booklets for feedback and verification of appropriate scoring. A further three day training program on the CLASS and ECERS-R was conducted each year, followed by clinical reliability testing of all researchers. Researchers who did not meet the reliable performance criteria (>80% fidelity on all observed items) did not proceed to collect data.
Analytic strategy
The study design was developed for multi-level modelling in which child, family, program and community levels of influence on children’s outcomes will be analysed. The basic analytic model for E4Kids is presented in Fig.1. Currently, analyses of the E4Kids data are underway, and will address the research questions of the study in the following manner:
E4Kids analytic model
- 1.
What features of ECEC provision promote children’s learning and social participation and define quality? Here analyses will focus on the comparison of data from Wave 1 (baseline) and Wave 2, and comparison of data from Wave 1 and Wave 3. Analyses will control for community, family and prior ECEC history, to compare child outcomes across and within program types. This will enable examination of the features of programs that best predict children’s outcomes. Features that consistently predict outcome, regardless of program type, will be identified.
- 2.
How does the ECEC experience affect children’s on-going development, educational attainment and social well-being? Children’s outcomes at NAPLAN testing will be modelled using community, family and program-level data, and will control for prior learning at Waves 2 and 3. Modelling will identify universal and context specific predictors of success.
- 3.
How do ECEC program inputs influence children’s developmental outcomes (educational attainment & social well-being)? Program data will act as an independent variable to identify outcomes that are significant predictors for children’s outcomes and child, family and community characteristics.
In addition, concurrent cost and price data will be analysed to enable the study to compare and contrast the change in child outcomes achieved through different programs (within and between program types). This will be achieved by using two distinct approaches that respond to the following questions:
- 4.
How cost effective are ECEC programs? Cost Effectiveness Analysis: Cost per significant developmental gain will be contrasted between programs. A program is cost-effective if it delivers desired effects at a lower cost per unit than alternative programs. A more robust understanding of the elements of quality will allow for comparison of individual program characteristics that enhance children’s development. There is also potential to raise program effectiveness with negligible impact to cost, through building enhanced understanding of the elements of program quality.
- 5.
What is the long-term return on investment in ECEC? Cost Benefit Analysis: It is possible to begin an analysis of ECEC programs in terms of their relative worth to the individual, public and society. Through statistical analysis, if an ongoing effect is found in achievement, then ECEC programs may play a role in deferring future remediation costs. By measuring the accrued benefits independently attributed to ECEC program participation, and contrasting them against a matched NPC, or low quality program group, a robust estimate of the net benefit to participant, family and community can be ascertained.
Ethics
This study is conducted under the approvals and protocols sanctioned by the University of Melbourne Human Research Ethics Committee (ID 0932660.2), and in accordance with linked approvals provided by the Victorian Government Department of Education and Training, the Queensland Department of Education and Training and the relevant Catholic Education Archdioceses. In accordance with the ethical approvals, formal written consent was obtained from each study participant, including the child’s main caregiver, the educators in programs, and school and service leaders. Verbal consent to take part in, or decline, each of the assessment activities was also obtained from each participant child, and all participants maintained the right to withdraw their participation at any time.
Availability of data and materials
In accordance with the terms and conditions agreed by the parties engaged in the study, data and study materials are owned by the University of Melbourne, and available the participating parties and researchers under license, for use in accordance with the approvals granted to the research team by the participants.
