• It maintains the parent's dynamic self-efficacy model comprising a self-efficacy value (1-7) for each of the tactics: 1 indicates a low level of self-efficacy, while 7 indicates a high level of self-efficacy.
• It represents the navigation history, a record of the navigation path the parent has taken through the educational material and a record of the tutoring tactics encountered.
• It maintains a record of the parent's name.
• It is dynamically updated during interaction to reflect the learner's current state.
• All affective states experienced by the child are logged, this allows for the provision of a history of the child's affective experiences.
• It also maintains a record of the child's current affective state and previous affective state.
• It maintains a record of the child's name.
• With the Suzuki violin domain, it maintains a record of the child's knowledge level in order to provide review suggestions tailored to the child's needs.
• Engage: the purpose of this is to attract the users' attention.
• Activate: the purpose of this stage is to allow the user to practice some newly acquired skill.
• Reinforce: the purpose of this stage is to reinforce the key message through reflection.
• Static self-efficacy—which comprised analysis of pre- and posttests, which were conducted prior to commencing the study and after their last interaction with P.A.C.T.
• Dynamic self-efficacy—which comprises a log of parents' self-efficacy scores for each tutoring tactic throughout the tutoring process, provides more precise insights into parents' self-efficacy. All self-efficacy scores were logged directly after the parent completed each activity. The instrument used for collecting the self-efficacy path values is illustrated in Fig. 19.
4.3.1 Static Self-Efficacy In terms of analysis of static self-efficacy, an increase in mean values across all seven tutoring tactics can be observed from pre- to posttest scores. This is illustrated in Table 1. A paired sample t-test was conducted, which showed a statistically significant increase at the level across mastery learning ( ), motivational game ( ), and repetition ( ). Additionally, positive reinforcement ( ) is approaching statistical significance. Only statistically significant p-values are listed in Table 1.
Expert demonstration, review, and tutoring variation showed no statistically significant increase between pre- and posttests scores. The lack of a significant increase in parents' self-efficacy across expert demonstration and tutoring variation is not altogether surprising considering the limited number of times these tutoring tactics were encountered. Fig. 6 demonstrates that of the 506 tutoring tactic suggestions P.A.C.T. made, 0.2 percent of these suggestions involved suggesting expert demonstration while 2 percent of the suggestions involved suggesting tutoring variation. This indicates that parents gained little experience in using these tactics, which may explain the lack of a statistically significant increase in their self-efficacy. The review tactic was suggested 21 percent of the time throughout the course of the study. This suggests that parents had the opportunity to gain significant experience in using the review tactic. Albeit, there was an increase in parents' self-efficacy between pre- and posttest scores for review, this increase was not statistically significant. Perhaps, the reason for this may be that on using P.A.C.T., parents gained a deeper understanding of the intricacies of the review tactic, and therefore, remained somewhat cautious in using it.
In summary, P.A.C.T. had a positive effect on parents' self-efficacy values as measured from pre- and posttest scores. For some tutoring tactics, namely, mastery learning, repetition, and motivational game, this increase was statistically significant.
4.3.2 Dynamic Self-Efficacy Since each parent submits a self-efficacy value after using each tutoring tactic, it is possible to determine a self-efficacy path (or dynamic self-efficacy model) for each parent across each tactic. This may be of benefit in understanding the effect on self-efficacy of supporting the home-tutoring process through the use of an intelligent tutoring system. As previously outlined, participants were randomly assigned to one of two groups. The first group received full support regardless of the self-efficacy value entered. The second group received adaptive support, where the level of support received was determined by their self-efficacy value for that tactic. Due to the adaptive nature of P.A.C.T. and the dynamic nature of the learning environment, each of the seven tutoring tactics were encountered with varying frequency (reader is directed to Fig. 6). As a result of this, it was decided to perform a pattern analysis of those tutoring tactics, which were most frequently encountered. These comprised mastery learning (23 percent), repetition (22 percent), and review (21 percent). Due to the nature of the data logged by the system, it is possible to plot individual self-efficacy paths. However, some interesting observations arise when self-efficacy paths are averaged across all participants. For the purpose of this study, the self-efficacy paths of participants receiving full support will first be analyzed, and subsequently, the self-efficacy paths of those who received adaptive support. This will provide insights into the effect of providing full versus adaptive support on parents' self-efficacy.
Figs. 7, 8, and 9 illustrate the self-efficacy paths for mastery learning, repetition, and review tutoring tactics for participants who received full-support when using P.A.C.T. The x-axis corresponds to the number of interactions with that tutoring tactic. The y-axis corresponds to parental self-efficacy and is measured using a value from one to seven. Each self-efficacy value illustrated in Fig. 7 is calculated based on the average self-efficacy value of all participants who received full support for that interaction, where all values are rounded to the nearest whole number. More specifically, the first value illustrated on the graph corresponds to the average submitted self-efficacy value of parents' first interaction with that tutoring tactic. In order to overcome the bias, a threshold was identified whereby averages were only included that comprised the self-efficacy values of a minimum of three participants. This was because the averages calculated, using self-efficacy values of less than three participants, may not be representative and may bias the results.
Fig. 7 shows a steady increase from 5 to 7 over the nine interactions. Fig. 8 shows no overall increase; however, some points on the graph show a marginal increase from a self-efficacy of 5 to 6. Fig. 9 shows an overall increase in self-efficacy from 5 to 6 with a peak in self-efficacy at interaction 7.
Albeit, the results suggest that overall P.A.C.T. had a positive effect on parental self-efficacy; it was expected that this effect would be more dramatic. However, qualitative data collected during interviews may provide further insight into these results. Following is a sample of the type of data pertaining to self-efficacy, which was collected during interviews with participants. Participant 1 states, "She was saying to me put in a 7. She thinks you have to put in the best one the whole time. She wants to be a part of the whole packet," while participant 2 states, "I wouldn't personally see any benefit and I don't do it with any thought there sometimes he wants to take over the mouse and he wants to click on the number he likes and my attitude is along as it is keeping him happy and it is keeping him involved and it is keeping him positive about the whole thing that's the biggest benefit for me."
Parents were asked how they felt about being asked to enter a self-efficacy value after using each of the tutoring tactics. The sample data presented in the previous paragraph indicates that the submitted self-efficacy value maybe the result of a suggestion by their child as opposed to a reflection of how confident they were feeling at that time. Participant 1 describes how her daughter wanted her to enter the highest self-efficacy value possible (7) as her daughter believes that it is important to enter the "best" number. This suggests an implication that the self-efficacy scale is graded, where seven is the best. Sarah reports that she does not see any personal benefit in submitting her self-efficacy value, and therefore, has developed somewhat of an ad hoc approach to it. Participant 2 states that the most important thing for her is keeping her son happy if that entails him choosing the value she is happy for that to happen, as due to parents receiving full support the value submitted has little relevance to them. This is understandable, as these participants received full support, the level of support remaining the same regardless of the submitted value.
In summary, few patterns have emerged from the self-efficacy paths of those parents who received full support from P.A.C.T. There was a slight increase in parents' self-efficacy when using review and a more significant increase in parents' self-efficacy when using mastery learning. Interestingly, all values (apart from the peak in mastery learning and review) across all paths lay in 5-6 efficacy level. However, conclusions may only be tentative as qualitative data suggests a somewhat ad hoc approach in submitting self-efficacy values due to a perception of its lack of relevance.
In terms of the effect of providing adaptive support on parental self-efficacy, Figs. 10, 11, and 12 show no clear patterns emerging. Fig. 10 illustrates the self-efficacy path for the mastery learning; however, the path is quite erratic with no clear increases. This is also the case for the repetition tutoring tactic detailed in Fig. 11. The self-efficacy path for the review tutoring tactic of those who received adaptive support is plotted in Fig. 12. This shows an overall increase in self-efficacy from 4 to 6. Again, the graph is erratic: the lowest value 3 was recorded on the 10th interaction, while the highest value 6 was recorded on the 13th interaction. All values lie between 3 and 6.
Qualitative data collected during interviews provide some insight into the aforementioned results. A sample of such data is now presented. Participant 3 states, " In the beginning I wasn't sure I was going one away from the top mark at one stage, when everything was a 7 you wouldn't get much to do, so on a couple of occasions I'd ease back and instead of doing all the 7s thinking I'm great we will come back and get our jobs to do." Participant 4 states, " That took me a while to manage because at the beginning I was saying I was very confident and I wasn't getting as much feedback as I did when I said I was less confident or that I hadn't a clue it seemed to work best when you said you hadn't an idea and look for suggestions rather than saying I'm very confident at doing this I'm very confident at doing the other where as if you suggested that you weren't as confident it gave you a few extra ideas." Participant 5 states, " Getting on better because going lower in scores gives you more help."
Interestingly, all sample data presented in the previous paragraph reports some level of experimentation with P.A.C.T. in an endeavor to receive the desired level of support. Qualitative data suggests that participants submitted self-efficacy values lower than desired in order to receive the desired level of support. To this end, the submitted value may be a representation of the level of support they desired as opposed to their perceived level of self-efficacy. For example, participant 3 reports of easing back in order to receive more "jobs" (more activity suggestions) from P.A.C.T, while participant 4's strategy is pretending that she "hadn't a clue." Participant 5 frames it as tricking the system into working for their benefit by submitting 1s. This is clearly illustrated in Fig. 13, where participant 5's self-efficacy values across mastery learning, repetition, and review quickly plummet. Certainly, it appears that there is a dichotomy between their perceived level of self-efficacy and the desired level of support; in so far as parents are entering high levels of self-efficacy but still desire high levels of support.
A comparison of self-efficacy paths of those participants who received full support and those participants who received adaptive support highlights the lack of emergent patterns. However, qualitative data indicates that the reason for this may vary between groups. Participants receiving full support indicated that there might have been a somewhat ad hoc approach in submitting self-efficacy values. On the other hand, participants receiving adaptive support indicated a need to submit self-efficacy values lower than desired in order to receive the desired level of support.
In summary, posttest scores indicate an increase in parental self-efficacy values across all seven tutoring tactics. However, on closer inspection, based on parents' self-efficacy paths, it may be that these values are not as true a reflection as initially expected. Parents receiving full support report that, on occasion, they allowed their child to select the value to submit. Parents receiving adaptive support identify the need to submit a value lower than their perceived self-efficacy value in order to receive the desired level of support. These results provide important insights in so far as they indicate dichotomy between parents' perceived self-efficacy and desired level of support. Second, it is clear that there is a need for a more subtle instrument for eliciting self-efficacy if it is to be used within adaptive educational systems.
5.3.1 Static Self-Efficacy Table 2 shows an increase in mean across all of the six tutoring tactics from pre- to posttest scores. This suggests an increase in parent self-efficacy on using the intelligent tutoring system. In addition, a paired sample t-test was conducted which identified a statistically significant increase at the level in the positive reinforcement ( ) and review ( ) tactics. Additionally, results based on the motivational game tactic were approaching statistical significance ( ). In Table 2, only statistically significant p-values are presented.
5.3.2 Dynamic Self-Efficacy with Additional Support As P.A.C.T. adapts the tutoring process based on the affective needs of the child, participants used each of the tutoring tactics a varying number of times. It can be observed from Fig. 15 that, of the 441 tactic suggestions that P.A.C.T. made, the review tutoring tactic was suggested most frequently (43 percent) followed by mastery learning (25 percent) and positive reinforcement (23 percent). Motivational Game (7 percent), tutoring variation (1 percent), and expert demonstration (1 percent) were suggested less often. Therefore, when analyzing parents' self-efficacy paths, we will concentrate on those tactics suggested more frequently, namely review, mastery learning, and positive reinforcement. First, we will look at the self-efficacy paths (or dynamic self-efficacy model) of those receiving full support. Subsequently, we will look at the self-efficacy path of those receiving adaptive support, and finally, we will provide some comparison.
Figs. 16, 17, and 18 illustrate the self-efficacy path for the review, mastery learning, and positive reinforcement tutoring tactics for participants who were provided with adaptive support when using P.A.C.T. Similar to study A, the reader is reminded that the x-axis corresponds to the number of interactions with the tutoring tactic; the number of interactions may vary for each tactic. The y-axis corresponds to self-efficacy and is measured using a value from 1 to 7. As with study A, in order to overcome bias, a threshold was identified whereby only values were included where they comprised the average of a minimum of three submitted self-efficacy values. The self-efficacy path is erratic across interactions. Fig. 16 shows a slight decrease in the review self-efficacy path from a self-efficacy value of 6 to 5. Fig. 17 illustrates the self-efficacy path for the mastery learning, and after the initial drop, a steady increase in self-efficacy values can be observed, save the last two values. Fig. 17 illustrates the self-efficacy path for the positive reinforcement tutoring tactic, where the graph appears a little more erratic than the previous two graphs.
In summary, few patterns have emerged from the self-efficacy paths of those parents who received full support from P.A.C.T. However, interestingly, all values across all paths lay in one of two efficacy levels, level 3-4 or level 5-6. The unstructured order of the efficacy paths may not be altogether surprising when one remembers that self-efficacy is task-specific. Albeit, every endeavor was made to encourage participants to submit a self-efficacy value corresponding to their confidence in using the tutoring tactic, this may not have occurred. Instead, it may be that participants submitted values based on their reaction to specific activities suggested by P.A.C.T. as opposed to their confidence in using the particular tactic.
Figs. 19, 20, and 21 illustrate the self-efficacy path for the review, mastery learning, and positive reinforcement tutoring tactics of those participants receiving adaptive support, where as with the other graphs, values are based on the self-efficacy value submitted by the parent and not on the requests for additional support. All three graphs show a steady increase in self-efficacy. Fig. 19 illustrates a significant increase in self-efficacy from 4 to 6 across the nine interactions.
Fig. 20 illustrates the self-efficacy path for the mastery learning tutoring tactic where, again, there is an overall steady increase in self-efficacy. There is one drop in self-efficacy, where at interaction 3, self-efficacy decreases from 5 to 4. At interaction 4, self-efficacy is on the increase once more with its value increasing from 4 to 5 for interaction 5. Finally, the self-efficacy path plotted in Fig. 21 represents the self-efficacy values for the positive reinforcement tutoring tactic. Similar to the mastery learning tactic with positive reinforcement, an overall increase in self-efficacy is illustrated. There is one drop in self-efficacy at the fifth interaction where self-efficacy decreases from 5 to 4. However, at interaction 6, self-efficacy increases once again with its value increasing from 4 to 5. Self-efficacy remains at a value of 5 for the remainder of the path.
Results indicate that overall there was an increase in self-efficacy across the review, positive reinforcement, and mastery learning tutoring tactics for those participants who received adaptive support. As described previously, P.A.C.T. adapts the level of support provided based on the self-efficacy value submitted by the parent for that tutoring tactic. However, in study B, P.A.C.T. also provided parents with an opportunity to request the next level of support. Interestingly, 63 percent of participants receiving adaptive support requested additional support from P.A.C.T. at some point throughout the study. This indicates that, on average, participants did not receive the desired level of support, which suggests a dichotomy between parental self-efficacy and desired level of support.
A number of patterns emerged from the data surrounding requests for additional support. Interestingly, further support was requested for four of the six tutoring tactics, namely, mastery learning, motivational game, positive reinforcement, and review. Based on all requests for additional support, the request for further support with review comprised 82 percent followed by mastery learning (9 percent), positive reinforcement (8 percent), and motivational game (1 percent) as illustrated in Fig. 22. This may not be altogether surprising as these were also the order for most frequently suggested tutoring tactics.
However, perhaps more interestingly, upon further inspection, results indicate that 56 percent of the time that the review tutoring tactic was suggested, participants asked for further support; 24 percent of the time that mastery learning was suggested, participants asked for further support; 23 percent of the time that positive reinforcement was suggested, participants asked for further support; and finally, 8 percent of the time that motivational game was suggested, participants asked for further support. This is illustrated in Fig. 23. Furthermore, based on all requests for additional support, 27 percent of requests involved parents requesting an explanation of the tutoring tactic (self-efficacy level 5-6), while 73 percent of requests involved parents requesting an example of the tutoring tactic (self-efficacy level 3-4). As participants can only request further support at the 3-4 self-efficacy level if they have previously received support at the 5-6 self-efficacy level, this suggests that participants may require an example in order to perform the task. Additionally, there may be a tendency to submit a self-efficacy value of 5-6 even if the level of support desired equates to a self-efficacy value of 3-4.
In terms of strategies used for requesting additional support, 31 percent of requests involved requesting further support at the 5-6 self-efficacy level and immediately requesting further support at the 3-4 self-efficacy level. This equates to asking P.A.C.T. for an explanation of the tutoring tactic and immediately asking for an example. Six percent of the requests involved only asking for further support at the 5-6 level, which corresponds to only asking for an explanation and not requiring an example.
Finally, 63 percent of requests involved asking for further support at the 3-4 self-efficacy level, which equates to P.A.C.T. suggesting an appropriate tutoring tactic and providing an explanation and the participant asking for an example. This suggests a dichotomy between parents' perceived level of self-efficacy and desired level of support, as 31 percent of requests involved participants submitting a self-efficacy value of 7 but desiring a level of support corresponding to a self-efficacy value of 3-4, which is a substantial difference.
In summary, results indicate an overall increase in self-efficacy between pre- and posttest. Self-efficacy paths of participants receiving adaptive support show an increase in self-efficacy. However, the same cannot be said for participants who did not receive adaptive support. On participants receiving adaptive support, results indicate a dichotomy between parents' perceived level of self-efficacy and desired level of support. More specifically, patterns have emerged which indicate that parents are entering efficacy values, which are too high for their desired level of support. Thirty-one percent of requests for further support were as a result of parents submitting a self-efficacy value of 7 but desiring a level of support corresponding to a self-efficacy value of 3-4. This may indicate a lack of understanding about the link between self-efficacy and ability, or indeed, an eagerness to appear confident in their ability leading to unwillingness to ask for assistance. Therefore, there is a need for future research into the design of a more subtle instrument for collecting self-efficacy values.
• O. Lahart is with the National College of Ireland, Mayor Street, IFSC, Dublin, Ireland. E-mail: firstname.lastname@example.org.
• D. Kelly is with the Centre for Research in IT in Education, Trinity College Dublin, National College of Ireland, Mayor Street, IFSC, Dublin 1, Ireland.
• B. Tangney is with the Centre for Research in IT in Education, Department of Computer Science, Trinity College Dublin, Dublin 2, Ireland.
Manuscript received 22 Dec. 2008; revised 13 Mar. 2009; accepted 3 Apr. 2009; published online 21 Apr. 2009.
For information on obtaining reprints of this article, please send e-mail to: email@example.com, and reference IEEECS Log Number TLTSI-2008-12-0118.
Digital Object Identifier no. 10.1109/TLT.2009.20.
Orla Lahart received the PhD degree from Trinity College Dublin. She is a member of the Research in Education and Learning Technology (REALT) Group at the National College of Ireland and Centre for Research in IT and Education (CRITE) at Trinity College Dublin, where she is an active researcher in adaptive education systems, tutoring strategies, and collaborative learning. She is a member of the IEEE.
Declan Kelly is with the Centre for Research in IT in Education, Trinity College Dublin. He is an active researcher in adaptive education systems, individual learning characteristics, and multiple intelligence. He is also working on the EDUCE adaptive education system that utilizes learning characteristics to provide an individualized learning environment. Previously, he was the Director of the Research in Education and Learning Technology (REALT) Group at the National College of Ireland. He was a local chair for the Adaptive Hypermedia Conference 2006 and a coauthor on the eLearning Research and Development Roadmap for Ireland, 2004. He is a member of the IEEE.
Brendan Tangney is a senior lecturer in the Department of Computer Science, founder and codirector of the Centre for Research in IT in Education (CRITE), and warden of Trinity Hall at the Trinity College Dublin, Ireland. CRITE's research covers areas ranging from the development of learning tools and environments to social outreach activities. His current research interests include the innovative use of ubiquitous technology to enhance the experience of learners. Particularly, his research has developed educational tools in the areas of adaptive hypermedia, music education, and learning tools for mobile devices. He has held visiting positions at the Universities of Sydney and Kyoto. He is a recipient of Trinity's Provost's Teaching Award for excellence and innovation in teaching and learning. He was a pedagogical advisor on MIT's Toy Symphony Project, is a member of the editorial review board for the AACE Journal of Computers in Mathematics and Science Teaching, and was a program chair for the 2007 Computer Assisted Learning Conference. He is a member of the IEEE.