M4Girls: Empowering Female Students

Posted by AnneryanHeatwole on Feb 23, 2010

The following is the executive summary of M4Girls, prepared by the Mindset Network and Neil Butcher and Associates, and reprinted here with permission from Mindset.

Introduction

The M4girls project is a partnership between Nokia, Mindset Network, and the Department of Education (North West Province/South Africa) to test the provision of educational content on a mobile phone platform to girl learners. The project targeted the development of Mathematics competencies in Grade Ten girl learners from underserved communities, and aimed to empower girl learners in the following ways:

Access to mathematics (as a pilot subject area and driven by priority areas in education in South Africa);

Exposure to a complementary platform of curriculum-aligned Mathematics content (Mindset content) on mobile phones; and

Exposure to technology in the form of mobile phones.

Two schools were selected as pilot schools for the project by the North West Department of Education in South Africa. The project involved issuing 20 girl learners in the two schools (ten in each school) with Nokia 6300 mobile phones that contained curriculum-aligned Mathematics content which was developed locally by Mindset Network. The content was presented in the form of easy-to-navigate games and videos that target the girl learners directly without requiring mediation or facilitation. Two games (Mathstermind and Fashion Empire), 47 videos, and two mobisodes (‘cartoon’ episodes applying Mathematics) were loaded onto the phones. 

The project ran from April to November, 2008.

Guiding Research Questions

The evaluation focused primarily on outcomes of the project and the central research questions included the following:

1. Did the Mathematics performance of girl learners in the Mobile Phone (MP) group improve after the start of the project?

2. Was there a change in attitude toward mathematics and education in general by learners in the MP group?

3. Was there a change in attitude towards and confidence in use of mobile phones and technology by girl learners in the MP group?

The research also addressed process issues by examining how the girl learners engaged with the phone and the content and actual usage of the various functionalities. This also included consideration of the relative success of different components of the content.

Methodology

The research design that guided evaluation of the project was a non-experimental pre-test post-test, matched comparison group design. A test group of 20 learners (ten learners from each school) was selected by educators to participate in the project. A different group of 20 girl learners, matched by the schools in terms of demographics and performance in Mathematics, comprised the comparison group. 

The following research questions guided the evaluation:

Was there a change in attitude toward mathematics and education in general by girl learners?

Was there a change in attitude and confidence in use of mobile phones and technology by girl learners?

What were the perceptions and reactions of girl learners to the educational content provided on the phones?

Did the Mathematics performance of girl learners in the mobile phone group improve after the start of the project?

Mindset researchers developed the research design, all instruments and collected all the data. Neil Butcher and Associates (NBA) were involved as external evaluators, reviewed some of the instruments and accompanied the Mindset researchers on one of the field visits in order to observe the data collection process. All data collected and analysed, as well as reports written during the course of the project were provided to NBA to assist in the preparation of this final report.

Context of the schools

Both schools are technical high schools situated in Rustenburg in the North West Province in South Africa. The schools are representative of peri-urban technical high schools accommodating over 1,200 learners and over 40 educators. The general level of education in the community is estimated between Grades Ten and Twelve, and the average income level ranges from R1,500 to R4,500 per month. Both schools are involved in three other projects which aim to improve Mathematics and science at the schools. School A is better equipped with technological resources. For example, it has four overhead projectors and seven printers, while School B has one projector and three printers. Nevertheless, educators at both schools tended to be familiar with using technologies like cell phones, television and DVD players and had a positive attitude towards using technology in teaching and learning. 

Profile of educators

Four educators, 2 from each school were interviewed at the start and end of the project. They all taught Mathematics or Mathematics Literacy. Two had an advanced certificate in education and 2 had education degrees. Experience ranged from 3-32 years. The educators reported having their own cell phones and felt comfortable in their ability to use them. From the baseline to the post-test educators demonstrated an increase in their perceptions of the advantages of using mobile phones for learning. However, a tension existed between the policies in schools regarding using mobile phones and implementing mobile phone projects.

Profile of learners

The baseline and post-project survey revealed that cell phones and television are the most common technologies used by learners. However, there did not appear to be any marked difference in the frequency of technology use between the comparison and mobile phone (MP) groups between the baseline and the post-project survey. In comparing technology use of the MP group before and after the project, it was noticed that Internet usage increased from 10% to 21% (perhaps due to learners using their cell phones to access the Internet). Learners also reported using the phones for a number of other functions including calling and sending and receiving messages, using the camera, calculator and video, reminders and the alarm functions, and using MiXit. 

Learners reported high comfort levels in using technology. Their comfort levels increased during the course of the project, notably in the use of the Internet, which increased from 50% in the baseline study to 71% (combined results for both the comparison and MP groups) in the post-project survey, with greater comfort levels noted in the MP group in the post-project survey (74% compared to 67% in the comparison group). Learners’ ratings on the importance of owning and being able to use technological equipment also increased during the course of the project – from 83% to 100% and 85% to 100% respectively. Whilst these responses reflect a growth in the extent to which ICT is valued by learners, this growth was noticed in both the MP and comparison group. Learners indicated that this was important because they are living in a technological age, technological skills are a prerequisite to securing jobs, and technology allows people to communicate and access information more easily.

All learners regarded school as being important, citing reasons such as enabling them to continue studying, get a job, and achieve a dream. Learners’ attitudes towards school, particularly towards studying, their future studies and their teachers were very positive and remained constant throughout the project. Learners’ feelings towards homework and schoolwork, whilst also rated highly in the baseline study, improved during the course of the project – from 85% to 97% for schoolwork, and from 90% to 97% for homework. However, fewer learners felt positive toward their friends, and these feelings worsened during the course of the project from 75% to 61% for all learners. For the MP group, the less positive feelings were more pronounced, with positive ratings of friends decreasing from 75% to 52%. Data from the interviews and focus group suggested that receiving a cell phone may have created jealousy between learners and this could possibly attribute to the lower positive feelings towards friends.

In the baseline survey, learners reported positive perceptions of Mathematics, with 84% rating their feelings about Mathematics positively, citing reasons such as it being a prerequisite for a job and enjoying doing Mathematics. The post-project survey revealed more positive attitudes towards Mathematics, with almost all (100% in the comparison and 95% in the MP group) learners reporting positive feelings about Mathematics, citing similar reasons such as Mathematics assisting them in obtaining a job, as well as posing a challenge.

Generally learners had a positive attitude towards using technologies for learning. Whilst a more positive attitude was noted during the post-project survey, no noteworthy difference was noted between the comparison and MP groups.

Results of Research Questions

1) Changes in performance in Mathematics

School performance records did not indicate any substantial difference in the performance in Mathematics between the Mobile Phone (MP)group and comparison group over the course of the year. Performance on the self-developed Maths Quiz indicated that the MP group performed slightly better than the comparison group.

Two sources of data were considered for assessing actual changes in performance, namely performance on a self-developed ‘Maths Quiz’ that the MP and comparison group completed both before and after the project, and performance records obtained from the schools for all grade 10 learners. Perceptions of whether the project improved performance was also asked of learners and educators in the interviews.

In terms of performance on the Maths Quiz, there was a slight improvement in the overall performance of learners, with the MP group performing better than the comparison group. There was some variation across the different items or questions in the quiz for task 1 and 2. In task 1, more improvement was noted for items on fractional multiplication, but poorer performance on simple addition, multiplication, and subtraction. In task 2, there was more improvement for items on coordinates and line equation, but poorer performance for items on parabolas and surface area. 

The performance records from both schools did not show any evidence for differences between the MP group and either the comparison group or the full cohort of grade 10 learners. This was true for the results of all assessments from the beginning of the year through to the final Mathematics examinations in November 2008.

It is essential to note that the educational content was in key priority areas and did not cover the full curriculum. It was thus used as needed by the girls. It was supplementary to the curriculum content in textbooks and provided in class and only lasted 6 months. One cannot truly attain a change in mathematics performance within 6 months, but rather longer periods of more consistent input over at least a 2 year period is often required. To this end, the school performance is a blunt measure of change. The maths quiz developed by Mindset is perhaps a truer reflection as it was developed with the content of the educational supplementary materials on the phones in mind. The suggestion of a better performance on the maths quiz points to a trend of positive impact that requires further investigation over a longer period of time and with a greater amount of content available.

Another key area to consider at this point is the manner in which the girls used the phones with the content. Interviews with the girls demonstrated that many girls used the phone to complete homework or prepare for classes the next day. The concept of a ‘teacher in my pocket’ raised by one girl is an illustration of this. The content was available when needed, but was not always needed. These indications of active, proactive responses to one’s own learning are encouraging. This is an area in which to develop mobile phone content – content available and possibly mediated to encourage active learning through the learner’s initiative. Greater levels of motivation and encouragement to stay on task, together with intrinsic motivators to learn, should all be considered for future studies. 

Mixed reactions to the efficacy of the phones were noted from the educators during interviews, with some indicating that the performance of some learners was improving (although they could not attribute this to the phone) and others suggesting that the phones did not affect performance. From both the focus groups and individual interviews with learners, the general consensus was that the project was improving their mathematical abilities, particularly the videos. 

2) Change in attitude to Mathematics and Education

Learners from both the comparison and MP group expressed positive attitudes regarding their schooling experience and education both before and after the project. Generally the groups’ responses did not differ although the following observations were noted. All learners, but especially the MP group, showed more negativity about school friends after the project. In addition, the MP group showed a growing valuing of the intrinsic value of school (i.e. learning for oneself) while the comparison group tended more towards extrinsic or instrumental valuing (i.e. studying to get the certificate). Although the differences were minimal, more participants in the comparison group than in the MP group indicated positive feelings about Mathematics and Mathematics tests, while more participants in the MP group expressed positive feelings about Mathematics Class. Preferences for learning activities remained fairly constant for both groups, however the comparison group in particular, showed an increase in learning Mathematics by practicing problems on one’s own. 

Attitudes were assessed by asking learners to rate their feelings towards various aspects of their schooling, future career, and mathematics. Overall, the responses from all learners were positive both before and after the project with the vast majority of the sample continuing to rate their feelings about school, studying, future studies, teachers, homework, and schoolwork as either ‘very good’ or ‘good’. Ratings of school friends were relatively lower to begin with, but also dropped over the course of the project and more so for the MP group than for the comparison group. Possible reasons for the less positive rating of school friends, gathered from the interviews and focus group discussion, is that friends can be ‘moody’, jealous, or untrustworthy. During interviews, difficulties with peers were mentioned by 32% of the learners in the MP group and was the most common complaint regarding school. Educators also reported that a number of learners in the comparison group expressed insecurity and feelings of inadequacy due to their placement in the comparison group. 

For the above reasons, along with issues of security, it is recommended that future studies do not provide mobile phones to learners or teachers in this manner. Besides the response from fellow learners, the feeling of being isolated, not involved in the project, not being able to direct the learner’s use of the phone and also not receiving a phone themselves resulted in negative feelings. Integrating professional development elements or methods of using mobile phones into a model of mobile phones in education may alleviate some of these feelings. 

Importance of school was rated highly by both groups of learners both before and after the project. In terms of their ratings of how hard they work at school, whilst all learners responded favourably, there was a greater improvement from the pre to the post testing for the MP group in particular. In terms of identifying what was most important at school, the two most commonly identified reasons were ‘learning for myself’ and ‘getting the certificate’ which represented intrinsic and extrinsic reasons respectively. For the MP group, the frequencies saw a drop in the identification of extrinsic reasons and a rise in intrinsic reasons from the pre to post testing. The pattern was reversed for the comparison group. Thus, on completion of the project, more participants in the MP group identified the intrinsic value of learning for oneself as the reason why school is important, whereas more of the comparison group pointed to the extrinsic value of receiving a certificate.  

Specifically regarding Mathematics, learners were asked to rate how much they liked the subject of Mathematics, Mathematics classes, and Mathematics tests. Overall, more participants tended to show more positive feelings in the post testing although no dramatic changes were evidenced. 

There was little evidence to suggest strong differences between the comparison and MP group, however the following marginal differences were noted. The MP group rated more favourably their feelings about Mathematics classes whilst the comparison group rated more favourably Mathematics as a subject and Mathematics tests. 

Regarding possible shifts in learning activities, both groups showed largely similar preferences before and after the project although an interesting finding was that for both groups, the preference for the learning activity of practicing Mathematics problems on one’s own increased after the project, and that the comparison group indicated a stronger preference than the MP group. 

3) Changes in attitude to technology, and technology in education

Positive ratings regarding attitudes to technology were evident in both groups before and after the project, however the results of the post-testing typically yielded more favourable ratings. Ratings of self-efficacy in the use of ICTs increased similarly for both groups. A similar increase in ratings of the importance of using technology was observed, with both groups ratings being almost equal for both groups. Many participants indicated that technology can facilitate learning and at least 74% of the MP group advocated issuing cell phones to more learners. 

Attitudes to technology were also assessed via rating scales and positive ratings for both groups before and after the project were evident. Participants were asked to rate how comfortable they felt in using various ICT equipment. Ratings were high before the project started (at least 69% of all learners rated themselves as ‘very comfortable’ or ‘comfortable’ for all ICTs listed including cell phones and the internet) but a slight improvement in ratings was still noted at the end of the project. Participants also rated how important they felt it was to own and use technological equipment and ratings increased. Particularly regarding the use of technology, both groups showed equally marked increases in positive ratings from 85% to 100% of the group. 

For both groups there was an increase in the number of participants who thought that the use of technology would help them learn. Both 70% of the comparison group and 70% of the MP group indicated that it would be beneficial at pre-testing, and at post-testing the percentages increased to 83% and 79% respectively.  More particularly, the feedback from the focus groups and interviews suggested that the MP group felt that the project had helped them with Mathematics and 74% felt that the phones should be issued to all learners. Some caution was raised from learners against widespread distribution of phones based primarily on the supposition that other learners may abuse the phone and not apply it for academic benefit. 

4) The learners usage of the phone and mobile content

The learners responded well to the cell phone technology and used a variety of the phone’s functionality for personal use. The content was also generally well received and additional recommendations for improvement were made. The videos were rated very favourably and learners referred to them for further explanation of mathematical concepts. The mobisodes were not well received and their educational value was not readily recognized. The Mathstermind game was extensively engaged with and the majority of learners reported enjoying the game and learning from it. Fewer learners engaged with the Fashion Empire and the strongest criticism was that it is too complicated and confusing. Several suggested improvements to the various forms of content were made by the learners.

The learners’ engagement with the phone was indicated by the usage data downloaded directly off the phone as well as the results of 4 focus groups (2 focus groups for the MP groups from each school) and individual interviews with the learners. It is noted at the outset that there were several limitations to the data downloaded off the phone (such as learner’s playing the tutorials instead of the game and logging in as different users) however the results do provide an indication of the learners’ general usage of the games. The results centre around usage of the cell phone generally and for personal use, and usage of the Mindset content (videos, mobisodes, Mathstermind and Fashion Empire)

PHONE USAGE: Spontaneous uptake of the cell phones was clearly demonstrated by all learners in the project and all learners had purchased and inserted a SIM card shortly after receiving the phone. Learners further explained in the interviews and focus groups that they used other functionalities on the phone include listening to music (reported by 94% of the group), taking pictures (89%), surfing the internet (63%) and using Bluetooth (52%). This was supported by observations made by researchers during collection of usage data as most phones had been ‘customised’ by learners. This demonstrates that the technology is accessible to the audience and easily adopted. 

VIDEOS: All learners watched the video with each learner watching on average 6 videos. Learners tended to view videos that were pertinent to the curriculum at a given time and to watch them on more than one occasion, and sometimes with friends. The feedback was overall very positive and all learners said in the interview that the videos helped them understand Mathematics better with 88% of the learners reaffirming this in the post-test questionnaires. Most learners, 63%, also felt that their friends had benefited from the videos. The learners explained that the advantages of the video is that the teacher in the video explains the content well and uses examples, and that videos can be watched repeatedly. A limitation noted was that learners cannot ask the teacher in the video for clarity if they don’t understand. It was also suggested by 37% of the group that the videos were too short and did not contain sufficient detail, and 11% felt that the picture was difficult to see clearly. Most of the learners suggested that they would download the videos if they were available on the internet and named several additional topics to address for both Mathematics and other subjects.

MOBISODES: Learners did not engage with the mobisodes to the same extent as the videos with 25% of the sample being able to describe the graffiti story and 50% the skateboard story. Generally the learners did not report any learning benefit from the mobisodes and 37% of the learners did not think they were supposed to learn about Mathematics from the mobisodes.

MATHSTERMIND GAME: Despite limitations in the data downloaded from the phone, it is clear that all learners play the Mathstermind game, and that all components of the game were played by at least some learners. The first section, arithmetic, was played by all learners. The lowest level reached was level 1 of 10 but on average learners reached level 7. The average time spent playing the game was 153 minutes, with the maximum time spent on the section being 615 minutes.  Usage of the inequalities section was lower, which is as to be expected as it follows sequentially from the arithmetic section. For this section, the average level reached was level 5 of 10, with learners playing the section for on average 83 minutes and one learner playing it the most for 532 minutes. Regarding the graphs section, the average level reached was level 3 of 10 with learners having played it for an average of 56 minutes. Ten learners reached the genius section, but of these, only 5 completed at least one question correctly. The amount of time spent on the genius section ranged from 2 to 115 minutes with the average being 8 minutes. 

The comments from the learners during the focus groups and interviews revealed that at least 89% of the learners felt that the game assisted them with Mathematics. It not only encouraged doing mental arithmetic but also allowed them to practice and better understand topics taught at school. The learners tended to describe the game as fun and challenging although some felt the game was too challenging and boring. Several suggestions were made for improvements to the game which included increasing the number of levels, providing hints, providing a ‘help button’, and allowing for sections of an equation to be completed at a time. 

FASHION EMPIRE GAME: As a group, the learners engaged with Fashion Empire less than Mathstermind. Over the course of the project, 9 learners actively played the game, with an additional 6 having played at least one of the tutorials. Of those that played the game, the sections most regularly played were Staff (with learners having hired on average 9 staff members), Pattern making (with learners having made on average 5 patterns), and Materials. No learners played the Loans section, and only one learner managed to store one container correctly in the Storage section. The game was thus not played to its full extent. Feedback from the interviews and focus group data suggested that the most enjoyable aspects of the game included designing the clothes, selling the clothes, the business aspect of the game and the budgeting. Almost half of the learners also suggested that the game could help them with Mathematics in terms of budgeting and measurement. However, at post testing, more than half of the sample did not rate the game positively, and many of the learners felt the game was too complicated and boring. Mid way through the project the game was updated to allow learners to play each section separately instead of requiring them to play each section in sequence, however in the individual interviews, 43% felt the change had not made a difference to the game. In addition to the complexity of the game which was experienced as confusing and frustrating, special mention was made of the storage component which learners appeared to struggle with the most.   

Discussion

As a precursor to the discussion, it is noted that substantial changes in overall performance from a six-month pilot is unlikely given the multitude of factors affecting an educational context (particularly since the project focused on supplementary as opposed to primary learning opportunities) and the short duration of the project. Despite the study’s inability to demonstrate significant improvement in school performance records for the MP group, several positive findings emerged from the study. Firstly, some evidence of improvement in mathematical ability for the MP group was found in the results of the self-developed ‘Maths Quiz’ which suggests an increase in competency. Secondly, the cell phone technology was readily consumed indicating that it presents a viable platform for the delivery of content. Thirdly, both learners receiving phones and those without reported more receptivity to the value of learning using technology in terms of the increased ratings of the value of technology and the possible learning benefits afforded by technology. That both groups gave higher ratings at the end of the project suggests, along with findings from interviews, that the content was shared with learners outside of the MP group and this, too, highlights the receptivity of learners to a mobile phone-based learning experiences.  Thirdly, there was some evidence to suggest that attitudes to education and Mathematics in particular increased over the course of the project, although this varied across the comparison and MP group regarding the particular aspect of education and it remains unclear how the project influenced perceptions for either group. A surprising finding was that negotiation of school friends emerged as an important factor to consider in such initiatives as learners in the comparison group struggled with not being selected as part of the MP group, and learners in the MP group struggled with coping with reactions of friends outside of the group. Such social interactions should be considered and managed as far as possible in future applications of the project. 

A fourth encouraging finding was that much of the content was well received by the learners and its educational potential recognized. The video content was most favourably rated and the videos functioned as quality, additional learning resources. The results suggest that mobisodes did not offer the same quality of learning experience and learners did not report benefiting from viewing them. Both the entertainment and educational value of the Mathstermind game was reported by learners indicating its value as a supplementary learning resource. The extent of use also suggests that the game targeted an appropriate range of competency. Fashion Empire appealed to a smaller audience but within this audience, the game was also experienced as entertaining and educational. The difficulty level or complexity of the game may account for the smaller audience. Several recommendations were made by learners to improve the quality and usability of the content which appears as Appendix A. The consumption of the material without mediation was demonstrated in the pilot as the learners were not shown what content was available and how to access it. While this points to the success of the project, increased mediation of use of the content is likely to improve not only how much content is accessed but also to enhance the learning experience. Mediating content thus presents a potential extension of the initiative in future applications. 

In conclusion, the results of the study motivate for future applications of the project as some evidence for positive outcomes was found. Future applications should however take into account recommendations contained in the report including refinement of the learning content, mediation of the material, incorporation of educators into the project, and management of the social impact of the project. 

Limitations

Several limitations regarding the pilot are noted. As previously mentioned, the validity of all the data downloaded from the phone could not be ascertained and hence firm inferences about actual usage, particularly over time, cannot be drawn. The data relating to the use of videos and mobisodes also relied on self-report and could not be verified. The post-project data set was also incomplete as many of the comparison participants had not been informed of the event and were absent on the day of data collection. In addition, two participants withdrew from the project. Thus fewer numbers in the comparison group undermined the findings of comparisons with the MP group. The findings noted above are also based on slight improvements in data and, whilst statistical significance testing was not completed, the differences do not appear to be substantial. The study was also unable to comparison for the effects of the Mindset Datacast solution which was installed at both schools at the same time as the M4girls project. 

Conclusion and Recommendations

The main aim of the M4girls project was to empower girl learners with curriculum-aligned Mathematics content on a mobile phone platform, and to determine whether this resulted in any significant change in their attitudes towards technology, school and an improvement in Mathematics. In addition, the value of the Mathematics content to learners was considered. Although changes were detected in attitudes towards technology on the part of learners during the project, the findings of this pilot was that there was no significant change in attitudes towards technology and school as a direct result of the technology used by the MP group in the project, given that post-project results for both the comparison and MP groups were generally quite similar. That, is whilst perceptions of technology were enhanced, these perceptions improved across both the MP and comparison group and the extent to which the M4girls project alone contributed to this is unclear.

It is further difficult to draw firm conclusions regarding changes in performance and attitude attributable exclusively to the project because:

The project ran over a short duration of 6 months only

The project focused on supplementary materials which only addressed only portions of the full curriculum

The Mindset datacast was installed at the same time as the cell phones were provided to learners. Consequently, this introduced additional variables to the study, making it more difficult to determine causality.

In addition, a full appreciation of the value of the project was limited in that phone usage data was collected at irregular intervals, and this did not allow for standardized, comparable data on phone and application usage to emerge. In addition, there was no objective measure for collecting data on how often the videos and Mobisodes were viewed. 

Nevertheless, it is clear that some benefits have been derived from learners’ (MP group) participation in the project. Whilst learners’ reports during the focus groups point to the phones mainly being used to listen to music, other general benefits of having access to a cell phone such as Internet access and communication were noted, and this made learners more confident and technologically savvy. During the interviews, it was observed that learners displayed a sense of accomplishment as they described proficiency in using various phone functions. Thus, exposure to technology in the form of cell phones was well received by the MP group. Positive attitudes towards using technology for learning, together with the reports of high cell phone usage (by both educators and learners), indicate the potential of using cell phones for e-learning or m-learning.

However, this also needs to be considered in a context of schools’ cell phone policies (i.e. whether learners are allowed to bring and use cell phones at school). In this case, although the school policies do not permit the use of cell phones, learners participating in the project did bring their phones to schools. Whilst this did not appear to be problematic in this project, future projects may need to consider such policies. 

In addition, whilst the results indicate that the use of cell phones may be beneficial in increasing educational opportunities in underserved areas, reports of jealousy from other learners and those in the comparison group, to the extent that one learner dropped out of the study because she was not given a cell phone, highlight a concern around the impact of introducing technology into an environment that confers status on a small group of people receiving the technology, even though this may not have been intended.

With regards to performance in Mathematics, concerns have been raised regarding the performance of both educators and learners. Although there are existing initiatives at both the schools to improve Mathematics, these efforts do not appear to have impacted on Mathematics performance, with the educators’ and learners’ performance on the Mindset Mathematic Quiz being low. In addition, this pilot project has also not demonstrated any significant impact on Mathematics performance, with there being no marked difference in the results of the MP and comparison groups over the duration of the project and both groups performing similarly in the baseline and post-project surveys (despite MP group learners reports that exposure to the M4girls material on the Nokia phone helped them to perform better at school and benefit academically). Whilst it is known that Mathematics requires greater attention (as highlighted by the National DoE initiatives), it appears that interventions at these schools have not had a visible impact. Thus, whilst there is a clear need for interventions focused on improving Mathematics, there is also a need to understand why existing interventions – despite their significant potential – are not resulting in visible impact.

There is a need to take cognizance of literature which indicates that poor performance in Mathematics and Science has been attributed to a shortage of qualified, confident and competent teachers at all levels of the schooling system. Furthermore, South Africa has recently undergone a process of curriculum transformation to which teachers are still adapting, and this further impacts on the potential for learners to succeed in these subjects. It should also be noted that both these schools reported large class sizes for Mathematics. Thus, whilst interventions focused on improving Mathematics are urgently needed, these issues may need to be more explicitly acknowledged and mitigated in future projects. 

As highlighted, some educators were not familiar with the content on the phones, and learners sometimes corrected educators showing them the videos to assist them in their teaching. This suggests that it might be valuable to give educators access to the content so that they can familiarize themselves with it, encourage learners to use it, and perhaps use it as a learning resource to be integrated with their existing teaching practices. In addition, given the poor performance of educators on the Mathematics quiz, and their reported ability and positive attitudes towards cell phones, it may be useful to load content onto educators’ phones as well. This can assist in providing a source of information as well as tackling the issue of a lack of technological resources at schools.

It is worth mentioning that, in general, learners’ attitudes towards school and Mathematics were positive. More positive perceptions towards Mathematics were noted in the post-project survey. However, whilst there was a more positive change in attitude towards Mathematics, this change was also noted for the comparison group. Therefore it is difficult to determine the precise cause of this enhanced positive attitude. It is also noteworthy that all learners (in both the MP and comparison group) are exploring career opportunities in the Mathematics field. Learners responded positively to the content in the Mathstermind game, reporting that it helped them with their Mathematics, challenged them and encouraged them to perform mental calculations. Fashion Empire was regarded less positively, and perceived to be too difficult and too confusing. Learners responded very positively to the videos and they were regarded as easy to understand, perhaps because they directly addressed the content areas that were covered in class. Thus, the learning opportunities for learners appear to have increased, particularly through use of the videos. The videos were perceived as helpful and explanatory as an alternate learning aid, to share the resource with other learners, and in some cases to even teach the teacher or confirm/reinforce the teachers’ explanations. However, extent of its use was difficult to determine as the video usage was not tracked during the collection of phone usage data. Most learners reported that they would download videos on different topics if they were available on the Internet. This suggests that the learners believe that continued exposure to Mindset videos would prove beneficial to them academically.

Thus, whilst the provision of cell phone technology may have improved learners’ proficiency in the use of various functions on a cell phone, it has not been possible to demonstrate unequivocally that provision of such technology will result in any learning gains in Mathematics. There was no marked difference in the attitude towards Mathematics and the performance of the mobile phone group compared to the comparison group over the course of the project, with Mathematics marks remaining poor. As has been noted, though, this does not mean that learning gains will not occur, although it does suggest that introduction of such technology needs to take place within a context where related problems (shortage of qualified teachers, large class sizes, and so on) before it is likely to have meaningful impact.

In order to improve the quality of the games, the following suggestions were made for Mathstermind:

Provide clearer instructions;

Include a ‘help key’ for signs or operators with which learners are not familiar;

Provide a whole equation to solve, or make provision to complete a number of steps before reaching the final answer in order to discourage guessing and encourage mental calculations; and

Provide hints or explanations of the mathematical concepts used in each level

For Fashion Empire, the following suggestions were made: 

Provide clearer instructions for both Mathstermind and Fashion Empire

Allow more freedom over the designing process;

Break down the stages into simpler steps; and

Simplify the ‘storage’ function

Recommendations for the video content include the following:

Given that the videos provide a more tangible benefit that learners found directly relevant to their school work provides an opportunity for larger scale dissemination, perhaps where the content can be downloaded from the Internet. 

There is also an opportunity to develop additional video content to cover other aspects of the curriculum.

The clarity of all content, particularly if it is harvested from existing content developed for television, should be evaluated to ensure that the images are clear given the size and colour limitations of mobile phone screens.

Future projects may consider incorporating interactivity to the video content. A ‘helpline’ to forward questions is one such possible form of interactivity.

Several learners suggested increasing the length and level of detail of the video content.

The following recommendations are therefore made for future projects:

The fact that the videos and Mathstermind game were used more by learners than the mobisodes and Fashion Empire suggests that content that is explicitly relevant to the curriculum may be better received, and therefore future projects may wish to focus on creating such content (as opposed to games like Fashion Empire where the content is more implicit and the mobisodes where the mathematical concepts were presented via stories). To have maximum effect, content provided should be regarded as relevant and appropriate by the users.

Future projects should run over a longer period if they are to provide meaningful opportunities to assess whether there are any changes in Mathematics performance (the short duration of the project clearly resulted in less visible changes in mathematics performance and attitudes. One cannot attain a clear pattern of educational attainment in less than a year).

In order to obtain a better understanding of the usefulness of the Mathematics content, it may be worthwhile to consider focusing on the provision of Mathematics content only, as opposed to the additional provision of cell phones (given that many learners already have access to cell phone technology, and the finding that learners being exposed to the Nokia phones resulted in the greater usage of other phone functionalities as opposed to the games and videos). However, it should also be acknowledged that low–end phones may not have technical features to download games.

Researchers need to play a more active role in negotiating sample selection, as this will decrease perceptions of favouritism from educators. In addition, it will reduce the risk of non-eligible learners (in this case mathematics literacy learners) being included in the project. Alternatively, full classes should be utilised.

There needs to be clear and consistent messaging about what will happen to technology provided after a project of this nature. Note that learners in the MP group still have the cell phones, even though they were informed that these will be taken away at the end of the project. It is further recommended that cell phones are not provided to a select few as this creates antagonism and ill feelings among learners. 

There is also a need for clear communication channels and responsibilities for all parties involved in the project. This was noted particularly for the data collection process, where there was a 60% response rate from the comparison group for the post-project survey and incomplete academic performance information from the schools.

Future research could consider a diversity of learners from rural and urban areas or perhaps only learners in an urban setting for the study. It might usefully include an option where learners can download curriculum-aligned content on their own cell phones via the Internet. This could allow for analysis of spontaneous uptake. If this approach is taken, together with downloading usage data to a computer, perhaps a different picture on usage may emerge. 

Phone usage data, including the use of videos and Mobisodes, needs to be tracked at regular intervals in order to allow for a more accurate picture of usage to emerge. There is also a need to determine how long learners play the Fashion Empire tutorial, as well as how many and for how long videos are accessed. The validity of the usage data for Mathstermind also needs to be revisited. It may also be useful to consider designing a programme to download the data automatically to a computer. In addition, data collection should be planned such that it is not done close to examination periods.

Tracking use of the internet, in terms of extent of surfing and the kinds of files accessed, would allow for a greater appreciation of learners’ preferences in the use of the internet and connectivity. 

Educators may need to be more involved in such projects. Further research could explore how this technology could be integrated within teaching and learning. For example, educators referring learners to a specific video as part of their homework, or using the games as a teaching tool – rather than this being an additional support used by learners as they wish to.

Basic Information
Organization involved in the project?: 
Project goals: 

The project aimed to:

  • Increase female students' aptitude with mathematics
  • Expose female students to technology through mobile phones
Brief description of the project: 

The M4girls project is a partnership between Nokia, Mindset Network, and the Department of Education (North West Province) to test the provision of educational content on a mobile phone platform to girl learners. The project targeted the development of Mathematics competencies in Grade Ten girl learners from underserved communities in South Africa, and aimed to empower girl learners. 

Target audience: 

The target audience is grade 10 girls in the North West Province of South Africa. 

Detailed Information
Length of Project (in months) : 
8
Status: 
Ended/Complete
What worked well? : 

Whilst learners’ reports during the focus groups point to the phones mainly being used to listen to music, other general benefits of having access to a cell phone such as Internet access and communication were noted, and this made learners more confident and technologically savvy. During the interviews, it was observed that learners displayed a sense of accomplishment as they described proficiency in using various phone functions. Thus, exposure to technology in the form of cell phones was well received by the MP group. Positive attitudes towards using technology for learning, together with the reports of high cell phone usage (by both educators and learners), indicate the potential of using cell phones for e-learning or m-learning.

 

What did not work? What were the challenges?: 

Although changes were detected in attitudes towards technology on the part of learners during the project, the findings of this pilot was that there was no significant change in attitudes towards technology and school as a direct result of the technology used by the MP group in the project, given that post-project results for both the comparison and MP groups were generally quite similar. That, is whilst perceptions of technology were enhanced, these perceptions improved across both the MP and comparison group and the extent to which the M4girls project alone contributed to this is unclear.

 

M4Girls: Empowering Female Students Locations

You need to upgrade your Flash Player
M4Girls: Empowering Female Students data sheet 5485 Views

The following is the executive summary of M4Girls, prepared by the Mindset Network and Neil Butcher and Associates, and reprinted here with permission from Mindset.

Introduction

The M4girls project is a partnership between Nokia, Mindset Network, and the Department of Education (North West Province/South Africa) to test the provision of educational content on a mobile phone platform to girl learners. The project targeted the development of Mathematics competencies in Grade Ten girl learners from underserved communities, and aimed to empower girl learners in the following ways:

Access to mathematics (as a pilot subject area and driven by priority areas in education in South Africa);

Exposure to a complementary platform of curriculum-aligned Mathematics content (Mindset content) on mobile phones; and

Exposure to technology in the form of mobile phones.

Two schools were selected as pilot schools for the project by the North West Department of Education in South Africa. The project involved issuing 20 girl learners in the two schools (ten in each school) with Nokia 6300 mobile phones that contained curriculum-aligned Mathematics content which was developed locally by Mindset Network. The content was presented in the form of easy-to-navigate games and videos that target the girl learners directly without requiring mediation or facilitation. Two games (Mathstermind and Fashion Empire), 47 videos, and two mobisodes (‘cartoon’ episodes applying Mathematics) were loaded onto the phones. 

The project ran from April to November, 2008.

Guiding Research Questions

The evaluation focused primarily on outcomes of the project and the central research questions included the following:

1. Did the Mathematics performance of girl learners in the Mobile Phone (MP) group improve after the start of the project?

2. Was there a change in attitude toward mathematics and education in general by learners in the MP group?

3. Was there a change in attitude towards and confidence in use of mobile phones and technology by girl learners in the MP group?

The research also addressed process issues by examining how the girl learners engaged with the phone and the content and actual usage of the various functionalities. This also included consideration of the relative success of different components of the content.

Methodology

The research design that guided evaluation of the project was a non-experimental pre-test post-test, matched comparison group design. A test group of 20 learners (ten learners from each school) was selected by educators to participate in the project. A different group of 20 girl learners, matched by the schools in terms of demographics and performance in Mathematics, comprised the comparison group. 

The following research questions guided the evaluation:

Was there a change in attitude toward mathematics and education in general by girl learners?

Was there a change in attitude and confidence in use of mobile phones and technology by girl learners?

What were the perceptions and reactions of girl learners to the educational content provided on the phones?

Did the Mathematics performance of girl learners in the mobile phone group improve after the start of the project?

Mindset researchers developed the research design, all instruments and collected all the data. Neil Butcher and Associates (NBA) were involved as external evaluators, reviewed some of the instruments and accompanied the Mindset researchers on one of the field visits in order to observe the data collection process. All data collected and analysed, as well as reports written during the course of the project were provided to NBA to assist in the preparation of this final report.

Context of the schools

Both schools are technical high schools situated in Rustenburg in the North West Province in South Africa. The schools are representative of peri-urban technical high schools accommodating over 1,200 learners and over 40 educators. The general level of education in the community is estimated between Grades Ten and Twelve, and the average income level ranges from R1,500 to R4,500 per month. Both schools are involved in three other projects which aim to improve Mathematics and science at the schools. School A is better equipped with technological resources. For example, it has four overhead projectors and seven printers, while School B has one projector and three printers. Nevertheless, educators at both schools tended to be familiar with using technologies like cell phones, television and DVD players and had a positive attitude towards using technology in teaching and learning. 

Profile of educators

Four educators, 2 from each school were interviewed at the start and end of the project. They all taught Mathematics or Mathematics Literacy. Two had an advanced certificate in education and 2 had education degrees. Experience ranged from 3-32 years. The educators reported having their own cell phones and felt comfortable in their ability to use them. From the baseline to the post-test educators demonstrated an increase in their perceptions of the advantages of using mobile phones for learning. However, a tension existed between the policies in schools regarding using mobile phones and implementing mobile phone projects.

Profile of learners

The baseline and post-project survey revealed that cell phones and television are the most common technologies used by learners. However, there did not appear to be any marked difference in the frequency of technology use between the comparison and mobile phone (MP) groups between the baseline and the post-project survey. In comparing technology use of the MP group before and after the project, it was noticed that Internet usage increased from 10% to 21% (perhaps due to learners using their cell phones to access the Internet). Learners also reported using the phones for a number of other functions including calling and sending and receiving messages, using the camera, calculator and video, reminders and the alarm functions, and using MiXit. 

Learners reported high comfort levels in using technology. Their comfort levels increased during the course of the project, notably in the use of the Internet, which increased from 50% in the baseline study to 71% (combined results for both the comparison and MP groups) in the post-project survey, with greater comfort levels noted in the MP group in the post-project survey (74% compared to 67% in the comparison group). Learners’ ratings on the importance of owning and being able to use technological equipment also increased during the course of the project – from 83% to 100% and 85% to 100% respectively. Whilst these responses reflect a growth in the extent to which ICT is valued by learners, this growth was noticed in both the MP and comparison group. Learners indicated that this was important because they are living in a technological age, technological skills are a prerequisite to securing jobs, and technology allows people to communicate and access information more easily.

All learners regarded school as being important, citing reasons such as enabling them to continue studying, get a job, and achieve a dream. Learners’ attitudes towards school, particularly towards studying, their future studies and their teachers were very positive and remained constant throughout the project. Learners’ feelings towards homework and schoolwork, whilst also rated highly in the baseline study, improved during the course of the project – from 85% to 97% for schoolwork, and from 90% to 97% for homework. However, fewer learners felt positive toward their friends, and these feelings worsened during the course of the project from 75% to 61% for all learners. For the MP group, the less positive feelings were more pronounced, with positive ratings of friends decreasing from 75% to 52%. Data from the interviews and focus group suggested that receiving a cell phone may have created jealousy between learners and this could possibly attribute to the lower positive feelings towards friends.

In the baseline survey, learners reported positive perceptions of Mathematics, with 84% rating their feelings about Mathematics positively, citing reasons such as it being a prerequisite for a job and enjoying doing Mathematics. The post-project survey revealed more positive attitudes towards Mathematics, with almost all (100% in the comparison and 95% in the MP group) learners reporting positive feelings about Mathematics, citing similar reasons such as Mathematics assisting them in obtaining a job, as well as posing a challenge.

Generally learners had a positive attitude towards using technologies for learning. Whilst a more positive attitude was noted during the post-project survey, no noteworthy difference was noted between the comparison and MP groups.

Results of Research Questions

1) Changes in performance in Mathematics

School performance records did not indicate any substantial difference in the performance in Mathematics between the Mobile Phone (MP)group and comparison group over the course of the year. Performance on the self-developed Maths Quiz indicated that the MP group performed slightly better than the comparison group.

Two sources of data were considered for assessing actual changes in performance, namely performance on a self-developed ‘Maths Quiz’ that the MP and comparison group completed both before and after the project, and performance records obtained from the schools for all grade 10 learners. Perceptions of whether the project improved performance was also asked of learners and educators in the interviews.

In terms of performance on the Maths Quiz, there was a slight improvement in the overall performance of learners, with the MP group performing better than the comparison group. There was some variation across the different items or questions in the quiz for task 1 and 2. In task 1, more improvement was noted for items on fractional multiplication, but poorer performance on simple addition, multiplication, and subtraction. In task 2, there was more improvement for items on coordinates and line equation, but poorer performance for items on parabolas and surface area. 

The performance records from both schools did not show any evidence for differences between the MP group and either the comparison group or the full cohort of grade 10 learners. This was true for the results of all assessments from the beginning of the year through to the final Mathematics examinations in November 2008.

It is essential to note that the educational content was in key priority areas and did not cover the full curriculum. It was thus used as needed by the girls. It was supplementary to the curriculum content in textbooks and provided in class and only lasted 6 months. One cannot truly attain a change in mathematics performance within 6 months, but rather longer periods of more consistent input over at least a 2 year period is often required. To this end, the school performance is a blunt measure of change. The maths quiz developed by Mindset is perhaps a truer reflection as it was developed with the content of the educational supplementary materials on the phones in mind. The suggestion of a better performance on the maths quiz points to a trend of positive impact that requires further investigation over a longer period of time and with a greater amount of content available.

Another key area to consider at this point is the manner in which the girls used the phones with the content. Interviews with the girls demonstrated that many girls used the phone to complete homework or prepare for classes the next day. The concept of a ‘teacher in my pocket’ raised by one girl is an illustration of this. The content was available when needed, but was not always needed. These indications of active, proactive responses to one’s own learning are encouraging. This is an area in which to develop mobile phone content – content available and possibly mediated to encourage active learning through the learner’s initiative. Greater levels of motivation and encouragement to stay on task, together with intrinsic motivators to learn, should all be considered for future studies. 

Mixed reactions to the efficacy of the phones were noted from the educators during interviews, with some indicating that the performance of some learners was improving (although they could not attribute this to the phone) and others suggesting that the phones did not affect performance. From both the focus groups and individual interviews with learners, the general consensus was that the project was improving their mathematical abilities, particularly the videos. 

2) Change in attitude to Mathematics and Education

Learners from both the comparison and MP group expressed positive attitudes regarding their schooling experience and education both before and after the project. Generally the groups’ responses did not differ although the following observations were noted. All learners, but especially the MP group, showed more negativity about school friends after the project. In addition, the MP group showed a growing valuing of the intrinsic value of school (i.e. learning for oneself) while the comparison group tended more towards extrinsic or instrumental valuing (i.e. studying to get the certificate). Although the differences were minimal, more participants in the comparison group than in the MP group indicated positive feelings about Mathematics and Mathematics tests, while more participants in the MP group expressed positive feelings about Mathematics Class. Preferences for learning activities remained fairly constant for both groups, however the comparison group in particular, showed an increase in learning Mathematics by practicing problems on one’s own. 

Attitudes were assessed by asking learners to rate their feelings towards various aspects of their schooling, future career, and mathematics. Overall, the responses from all learners were positive both before and after the project with the vast majority of the sample continuing to rate their feelings about school, studying, future studies, teachers, homework, and schoolwork as either ‘very good’ or ‘good’. Ratings of school friends were relatively lower to begin with, but also dropped over the course of the project and more so for the MP group than for the comparison group. Possible reasons for the less positive rating of school friends, gathered from the interviews and focus group discussion, is that friends can be ‘moody’, jealous, or untrustworthy. During interviews, difficulties with peers were mentioned by 32% of the learners in the MP group and was the most common complaint regarding school. Educators also reported that a number of learners in the comparison group expressed insecurity and feelings of inadequacy due to their placement in the comparison group. 

For the above reasons, along with issues of security, it is recommended that future studies do not provide mobile phones to learners or teachers in this manner. Besides the response from fellow learners, the feeling of being isolated, not involved in the project, not being able to direct the learner’s use of the phone and also not receiving a phone themselves resulted in negative feelings. Integrating professional development elements or methods of using mobile phones into a model of mobile phones in education may alleviate some of these feelings. 

Importance of school was rated highly by both groups of learners both before and after the project. In terms of their ratings of how hard they work at school, whilst all learners responded favourably, there was a greater improvement from the pre to the post testing for the MP group in particular. In terms of identifying what was most important at school, the two most commonly identified reasons were ‘learning for myself’ and ‘getting the certificate’ which represented intrinsic and extrinsic reasons respectively. For the MP group, the frequencies saw a drop in the identification of extrinsic reasons and a rise in intrinsic reasons from the pre to post testing. The pattern was reversed for the comparison group. Thus, on completion of the project, more participants in the MP group identified the intrinsic value of learning for oneself as the reason why school is important, whereas more of the comparison group pointed to the extrinsic value of receiving a certificate.  

Specifically regarding Mathematics, learners were asked to rate how much they liked the subject of Mathematics, Mathematics classes, and Mathematics tests. Overall, more participants tended to show more positive feelings in the post testing although no dramatic changes were evidenced. 

There was little evidence to suggest strong differences between the comparison and MP group, however the following marginal differences were noted. The MP group rated more favourably their feelings about Mathematics classes whilst the comparison group rated more favourably Mathematics as a subject and Mathematics tests. 

Regarding possible shifts in learning activities, both groups showed largely similar preferences before and after the project although an interesting finding was that for both groups, the preference for the learning activity of practicing Mathematics problems on one’s own increased after the project, and that the comparison group indicated a stronger preference than the MP group. 

3) Changes in attitude to technology, and technology in education

Positive ratings regarding attitudes to technology were evident in both groups before and after the project, however the results of the post-testing typically yielded more favourable ratings. Ratings of self-efficacy in the use of ICTs increased similarly for both groups. A similar increase in ratings of the importance of using technology was observed, with both groups ratings being almost equal for both groups. Many participants indicated that technology can facilitate learning and at least 74% of the MP group advocated issuing cell phones to more learners. 

Attitudes to technology were also assessed via rating scales and positive ratings for both groups before and after the project were evident. Participants were asked to rate how comfortable they felt in using various ICT equipment. Ratings were high before the project started (at least 69% of all learners rated themselves as ‘very comfortable’ or ‘comfortable’ for all ICTs listed including cell phones and the internet) but a slight improvement in ratings was still noted at the end of the project. Participants also rated how important they felt it was to own and use technological equipment and ratings increased. Particularly regarding the use of technology, both groups showed equally marked increases in positive ratings from 85% to 100% of the group. 

For both groups there was an increase in the number of participants who thought that the use of technology would help them learn. Both 70% of the comparison group and 70% of the MP group indicated that it would be beneficial at pre-testing, and at post-testing the percentages increased to 83% and 79% respectively.  More particularly, the feedback from the focus groups and interviews suggested that the MP group felt that the project had helped them with Mathematics and 74% felt that the phones should be issued to all learners. Some caution was raised from learners against widespread distribution of phones based primarily on the supposition that other learners may abuse the phone and not apply it for academic benefit. 

4) The learners usage of the phone and mobile content

The learners responded well to the cell phone technology and used a variety of the phone’s functionality for personal use. The content was also generally well received and additional recommendations for improvement were made. The videos were rated very favourably and learners referred to them for further explanation of mathematical concepts. The mobisodes were not well received and their educational value was not readily recognized. The Mathstermind game was extensively engaged with and the majority of learners reported enjoying the game and learning from it. Fewer learners engaged with the Fashion Empire and the strongest criticism was that it is too complicated and confusing. Several suggested improvements to the various forms of content were made by the learners.

The learners’ engagement with the phone was indicated by the usage data downloaded directly off the phone as well as the results of 4 focus groups (2 focus groups for the MP groups from each school) and individual interviews with the learners. It is noted at the outset that there were several limitations to the data downloaded off the phone (such as learner’s playing the tutorials instead of the game and logging in as different users) however the results do provide an indication of the learners’ general usage of the games. The results centre around usage of the cell phone generally and for personal use, and usage of the Mindset content (videos, mobisodes, Mathstermind and Fashion Empire)

PHONE USAGE: Spontaneous uptake of the cell phones was clearly demonstrated by all learners in the project and all learners had purchased and inserted a SIM card shortly after receiving the phone. Learners further explained in the interviews and focus groups that they used other functionalities on the phone include listening to music (reported by 94% of the group), taking pictures (89%), surfing the internet (63%) and using Bluetooth (52%). This was supported by observations made by researchers during collection of usage data as most phones had been ‘customised’ by learners. This demonstrates that the technology is accessible to the audience and easily adopted. 

VIDEOS: All learners watched the video with each learner watching on average 6 videos. Learners tended to view videos that were pertinent to the curriculum at a given time and to watch them on more than one occasion, and sometimes with friends. The feedback was overall very positive and all learners said in the interview that the videos helped them understand Mathematics better with 88% of the learners reaffirming this in the post-test questionnaires. Most learners, 63%, also felt that their friends had benefited from the videos. The learners explained that the advantages of the video is that the teacher in the video explains the content well and uses examples, and that videos can be watched repeatedly. A limitation noted was that learners cannot ask the teacher in the video for clarity if they don’t understand. It was also suggested by 37% of the group that the videos were too short and did not contain sufficient detail, and 11% felt that the picture was difficult to see clearly. Most of the learners suggested that they would download the videos if they were available on the internet and named several additional topics to address for both Mathematics and other subjects.

MOBISODES: Learners did not engage with the mobisodes to the same extent as the videos with 25% of the sample being able to describe the graffiti story and 50% the skateboard story. Generally the learners did not report any learning benefit from the mobisodes and 37% of the learners did not think they were supposed to learn about Mathematics from the mobisodes.

MATHSTERMIND GAME: Despite limitations in the data downloaded from the phone, it is clear that all learners play the Mathstermind game, and that all components of the game were played by at least some learners. The first section, arithmetic, was played by all learners. The lowest level reached was level 1 of 10 but on average learners reached level 7. The average time spent playing the game was 153 minutes, with the maximum time spent on the section being 615 minutes.  Usage of the inequalities section was lower, which is as to be expected as it follows sequentially from the arithmetic section. For this section, the average level reached was level 5 of 10, with learners playing the section for on average 83 minutes and one learner playing it the most for 532 minutes. Regarding the graphs section, the average level reached was level 3 of 10 with learners having played it for an average of 56 minutes. Ten learners reached the genius section, but of these, only 5 completed at least one question correctly. The amount of time spent on the genius section ranged from 2 to 115 minutes with the average being 8 minutes. 

The comments from the learners during the focus groups and interviews revealed that at least 89% of the learners felt that the game assisted them with Mathematics. It not only encouraged doing mental arithmetic but also allowed them to practice and better understand topics taught at school. The learners tended to describe the game as fun and challenging although some felt the game was too challenging and boring. Several suggestions were made for improvements to the game which included increasing the number of levels, providing hints, providing a ‘help button’, and allowing for sections of an equation to be completed at a time. 

FASHION EMPIRE GAME: As a group, the learners engaged with Fashion Empire less than Mathstermind. Over the course of the project, 9 learners actively played the game, with an additional 6 having played at least one of the tutorials. Of those that played the game, the sections most regularly played were Staff (with learners having hired on average 9 staff members), Pattern making (with learners having made on average 5 patterns), and Materials. No learners played the Loans section, and only one learner managed to store one container correctly in the Storage section. The game was thus not played to its full extent. Feedback from the interviews and focus group data suggested that the most enjoyable aspects of the game included designing the clothes, selling the clothes, the business aspect of the game and the budgeting. Almost half of the learners also suggested that the game could help them with Mathematics in terms of budgeting and measurement. However, at post testing, more than half of the sample did not rate the game positively, and many of the learners felt the game was too complicated and boring. Mid way through the project the game was updated to allow learners to play each section separately instead of requiring them to play each section in sequence, however in the individual interviews, 43% felt the change had not made a difference to the game. In addition to the complexity of the game which was experienced as confusing and frustrating, special mention was made of the storage component which learners appeared to struggle with the most.   

Discussion

As a precursor to the discussion, it is noted that substantial changes in overall performance from a six-month pilot is unlikely given the multitude of factors affecting an educational context (particularly since the project focused on supplementary as opposed to primary learning opportunities) and the short duration of the project. Despite the study’s inability to demonstrate significant improvement in school performance records for the MP group, several positive findings emerged from the study. Firstly, some evidence of improvement in mathematical ability for the MP group was found in the results of the self-developed ‘Maths Quiz’ which suggests an increase in competency. Secondly, the cell phone technology was readily consumed indicating that it presents a viable platform for the delivery of content. Thirdly, both learners receiving phones and those without reported more receptivity to the value of learning using technology in terms of the increased ratings of the value of technology and the possible learning benefits afforded by technology. That both groups gave higher ratings at the end of the project suggests, along with findings from interviews, that the content was shared with learners outside of the MP group and this, too, highlights the receptivity of learners to a mobile phone-based learning experiences.  Thirdly, there was some evidence to suggest that attitudes to education and Mathematics in particular increased over the course of the project, although this varied across the comparison and MP group regarding the particular aspect of education and it remains unclear how the project influenced perceptions for either group. A surprising finding was that negotiation of school friends emerged as an important factor to consider in such initiatives as learners in the comparison group struggled with not being selected as part of the MP group, and learners in the MP group struggled with coping with reactions of friends outside of the group. Such social interactions should be considered and managed as far as possible in future applications of the project. 

A fourth encouraging finding was that much of the content was well received by the learners and its educational potential recognized. The video content was most favourably rated and the videos functioned as quality, additional learning resources. The results suggest that mobisodes did not offer the same quality of learning experience and learners did not report benefiting from viewing them. Both the entertainment and educational value of the Mathstermind game was reported by learners indicating its value as a supplementary learning resource. The extent of use also suggests that the game targeted an appropriate range of competency. Fashion Empire appealed to a smaller audience but within this audience, the game was also experienced as entertaining and educational. The difficulty level or complexity of the game may account for the smaller audience. Several recommendations were made by learners to improve the quality and usability of the content which appears as Appendix A. The consumption of the material without mediation was demonstrated in the pilot as the learners were not shown what content was available and how to access it. While this points to the success of the project, increased mediation of use of the content is likely to improve not only how much content is accessed but also to enhance the learning experience. Mediating content thus presents a potential extension of the initiative in future applications. 

In conclusion, the results of the study motivate for future applications of the project as some evidence for positive outcomes was found. Future applications should however take into account recommendations contained in the report including refinement of the learning content, mediation of the material, incorporation of educators into the project, and management of the social impact of the project. 

Limitations

Several limitations regarding the pilot are noted. As previously mentioned, the validity of all the data downloaded from the phone could not be ascertained and hence firm inferences about actual usage, particularly over time, cannot be drawn. The data relating to the use of videos and mobisodes also relied on self-report and could not be verified. The post-project data set was also incomplete as many of the comparison participants had not been informed of the event and were absent on the day of data collection. In addition, two participants withdrew from the project. Thus fewer numbers in the comparison group undermined the findings of comparisons with the MP group. The findings noted above are also based on slight improvements in data and, whilst statistical significance testing was not completed, the differences do not appear to be substantial. The study was also unable to comparison for the effects of the Mindset Datacast solution which was installed at both schools at the same time as the M4girls project. 

Conclusion and Recommendations

The main aim of the M4girls project was to empower girl learners with curriculum-aligned Mathematics content on a mobile phone platform, and to determine whether this resulted in any significant change in their attitudes towards technology, school and an improvement in Mathematics. In addition, the value of the Mathematics content to learners was considered. Although changes were detected in attitudes towards technology on the part of learners during the project, the findings of this pilot was that there was no significant change in attitudes towards technology and school as a direct result of the technology used by the MP group in the project, given that post-project results for both the comparison and MP groups were generally quite similar. That, is whilst perceptions of technology were enhanced, these perceptions improved across both the MP and comparison group and the extent to which the M4girls project alone contributed to this is unclear.

It is further difficult to draw firm conclusions regarding changes in performance and attitude attributable exclusively to the project because:

The project ran over a short duration of 6 months only

The project focused on supplementary materials which only addressed only portions of the full curriculum

The Mindset datacast was installed at the same time as the cell phones were provided to learners. Consequently, this introduced additional variables to the study, making it more difficult to determine causality.

In addition, a full appreciation of the value of the project was limited in that phone usage data was collected at irregular intervals, and this did not allow for standardized, comparable data on phone and application usage to emerge. In addition, there was no objective measure for collecting data on how often the videos and Mobisodes were viewed. 

Nevertheless, it is clear that some benefits have been derived from learners’ (MP group) participation in the project. Whilst learners’ reports during the focus groups point to the phones mainly being used to listen to music, other general benefits of having access to a cell phone such as Internet access and communication were noted, and this made learners more confident and technologically savvy. During the interviews, it was observed that learners displayed a sense of accomplishment as they described proficiency in using various phone functions. Thus, exposure to technology in the form of cell phones was well received by the MP group. Positive attitudes towards using technology for learning, together with the reports of high cell phone usage (by both educators and learners), indicate the potential of using cell phones for e-learning or m-learning.

However, this also needs to be considered in a context of schools’ cell phone policies (i.e. whether learners are allowed to bring and use cell phones at school). In this case, although the school policies do not permit the use of cell phones, learners participating in the project did bring their phones to schools. Whilst this did not appear to be problematic in this project, future projects may need to consider such policies. 

In addition, whilst the results indicate that the use of cell phones may be beneficial in increasing educational opportunities in underserved areas, reports of jealousy from other learners and those in the comparison group, to the extent that one learner dropped out of the study because she was not given a cell phone, highlight a concern around the impact of introducing technology into an environment that confers status on a small group of people receiving the technology, even though this may not have been intended.

With regards to performance in Mathematics, concerns have been raised regarding the performance of both educators and learners. Although there are existing initiatives at both the schools to improve Mathematics, these efforts do not appear to have impacted on Mathematics performance, with the educators’ and learners’ performance on the Mindset Mathematic Quiz being low. In addition, this pilot project has also not demonstrated any significant impact on Mathematics performance, with there being no marked difference in the results of the MP and comparison groups over the duration of the project and both groups performing similarly in the baseline and post-project surveys (despite MP group learners reports that exposure to the M4girls material on the Nokia phone helped them to perform better at school and benefit academically). Whilst it is known that Mathematics requires greater attention (as highlighted by the National DoE initiatives), it appears that interventions at these schools have not had a visible impact. Thus, whilst there is a clear need for interventions focused on improving Mathematics, there is also a need to understand why existing interventions – despite their significant potential – are not resulting in visible impact.

There is a need to take cognizance of literature which indicates that poor performance in Mathematics and Science has been attributed to a shortage of qualified, confident and competent teachers at all levels of the schooling system. Furthermore, South Africa has recently undergone a process of curriculum transformation to which teachers are still adapting, and this further impacts on the potential for learners to succeed in these subjects. It should also be noted that both these schools reported large class sizes for Mathematics. Thus, whilst interventions focused on improving Mathematics are urgently needed, these issues may need to be more explicitly acknowledged and mitigated in future projects. 

As highlighted, some educators were not familiar with the content on the phones, and learners sometimes corrected educators showing them the videos to assist them in their teaching. This suggests that it might be valuable to give educators access to the content so that they can familiarize themselves with it, encourage learners to use it, and perhaps use it as a learning resource to be integrated with their existing teaching practices. In addition, given the poor performance of educators on the Mathematics quiz, and their reported ability and positive attitudes towards cell phones, it may be useful to load content onto educators’ phones as well. This can assist in providing a source of information as well as tackling the issue of a lack of technological resources at schools.

It is worth mentioning that, in general, learners’ attitudes towards school and Mathematics were positive. More positive perceptions towards Mathematics were noted in the post-project survey. However, whilst there was a more positive change in attitude towards Mathematics, this change was also noted for the comparison group. Therefore it is difficult to determine the precise cause of this enhanced positive attitude. It is also noteworthy that all learners (in both the MP and comparison group) are exploring career opportunities in the Mathematics field. Learners responded positively to the content in the Mathstermind game, reporting that it helped them with their Mathematics, challenged them and encouraged them to perform mental calculations. Fashion Empire was regarded less positively, and perceived to be too difficult and too confusing. Learners responded very positively to the videos and they were regarded as easy to understand, perhaps because they directly addressed the content areas that were covered in class. Thus, the learning opportunities for learners appear to have increased, particularly through use of the videos. The videos were perceived as helpful and explanatory as an alternate learning aid, to share the resource with other learners, and in some cases to even teach the teacher or confirm/reinforce the teachers’ explanations. However, extent of its use was difficult to determine as the video usage was not tracked during the collection of phone usage data. Most learners reported that they would download videos on different topics if they were available on the Internet. This suggests that the learners believe that continued exposure to Mindset videos would prove beneficial to them academically.

Thus, whilst the provision of cell phone technology may have improved learners’ proficiency in the use of various functions on a cell phone, it has not been possible to demonstrate unequivocally that provision of such technology will result in any learning gains in Mathematics. There was no marked difference in the attitude towards Mathematics and the performance of the mobile phone group compared to the comparison group over the course of the project, with Mathematics marks remaining poor. As has been noted, though, this does not mean that learning gains will not occur, although it does suggest that introduction of such technology needs to take place within a context where related problems (shortage of qualified teachers, large class sizes, and so on) before it is likely to have meaningful impact.

In order to improve the quality of the games, the following suggestions were made for Mathstermind:

Provide clearer instructions;

Include a ‘help key’ for signs or operators with which learners are not familiar;

Provide a whole equation to solve, or make provision to complete a number of steps before reaching the final answer in order to discourage guessing and encourage mental calculations; and

Provide hints or explanations of the mathematical concepts used in each level

For Fashion Empire, the following suggestions were made: 

Provide clearer instructions for both Mathstermind and Fashion Empire

Allow more freedom over the designing process;

Break down the stages into simpler steps; and

Simplify the ‘storage’ function

Recommendations for the video content include the following:

Given that the videos provide a more tangible benefit that learners found directly relevant to their school work provides an opportunity for larger scale dissemination, perhaps where the content can be downloaded from the Internet. 

There is also an opportunity to develop additional video content to cover other aspects of the curriculum.

The clarity of all content, particularly if it is harvested from existing content developed for television, should be evaluated to ensure that the images are clear given the size and colour limitations of mobile phone screens.

Future projects may consider incorporating interactivity to the video content. A ‘helpline’ to forward questions is one such possible form of interactivity.

Several learners suggested increasing the length and level of detail of the video content.

The following recommendations are therefore made for future projects:

The fact that the videos and Mathstermind game were used more by learners than the mobisodes and Fashion Empire suggests that content that is explicitly relevant to the curriculum may be better received, and therefore future projects may wish to focus on creating such content (as opposed to games like Fashion Empire where the content is more implicit and the mobisodes where the mathematical concepts were presented via stories). To have maximum effect, content provided should be regarded as relevant and appropriate by the users.

Future projects should run over a longer period if they are to provide meaningful opportunities to assess whether there are any changes in Mathematics performance (the short duration of the project clearly resulted in less visible changes in mathematics performance and attitudes. One cannot attain a clear pattern of educational attainment in less than a year).

In order to obtain a better understanding of the usefulness of the Mathematics content, it may be worthwhile to consider focusing on the provision of Mathematics content only, as opposed to the additional provision of cell phones (given that many learners already have access to cell phone technology, and the finding that learners being exposed to the Nokia phones resulted in the greater usage of other phone functionalities as opposed to the games and videos). However, it should also be acknowledged that low–end phones may not have technical features to download games.

Researchers need to play a more active role in negotiating sample selection, as this will decrease perceptions of favouritism from educators. In addition, it will reduce the risk of non-eligible learners (in this case mathematics literacy learners) being included in the project. Alternatively, full classes should be utilised.

There needs to be clear and consistent messaging about what will happen to technology provided after a project of this nature. Note that learners in the MP group still have the cell phones, even though they were informed that these will be taken away at the end of the project. It is further recommended that cell phones are not provided to a select few as this creates antagonism and ill feelings among learners. 

There is also a need for clear communication channels and responsibilities for all parties involved in the project. This was noted particularly for the data collection process, where there was a 60% response rate from the comparison group for the post-project survey and incomplete academic performance information from the schools.

Future research could consider a diversity of learners from rural and urban areas or perhaps only learners in an urban setting for the study. It might usefully include an option where learners can download curriculum-aligned content on their own cell phones via the Internet. This could allow for analysis of spontaneous uptake. If this approach is taken, together with downloading usage data to a computer, perhaps a different picture on usage may emerge. 

Phone usage data, including the use of videos and Mobisodes, needs to be tracked at regular intervals in order to allow for a more accurate picture of usage to emerge. There is also a need to determine how long learners play the Fashion Empire tutorial, as well as how many and for how long videos are accessed. The validity of the usage data for Mathstermind also needs to be revisited. It may also be useful to consider designing a programme to download the data automatically to a computer. In addition, data collection should be planned such that it is not done close to examination periods.

Tracking use of the internet, in terms of extent of surfing and the kinds of files accessed, would allow for a greater appreciation of learners’ preferences in the use of the internet and connectivity. 

Educators may need to be more involved in such projects. Further research could explore how this technology could be integrated within teaching and learning. For example, educators referring learners to a specific video as part of their homework, or using the games as a teaching tool – rather than this being an additional support used by learners as they wish to.

Basic Information
Organization involved in the project?: 
Project goals: 

The project aimed to:

  • Increase female students' aptitude with mathematics
  • Expose female students to technology through mobile phones
Brief description of the project: 

The M4girls project is a partnership between Nokia, Mindset Network, and the Department of Education (North West Province) to test the provision of educational content on a mobile phone platform to girl learners. The project targeted the development of Mathematics competencies in Grade Ten girl learners from underserved communities in South Africa, and aimed to empower girl learners. 

Target audience: 

The target audience is grade 10 girls in the North West Province of South Africa. 

Detailed Information
Length of Project (in months) : 
8
Status: 
Ended/Complete
What worked well? : 

Whilst learners’ reports during the focus groups point to the phones mainly being used to listen to music, other general benefits of having access to a cell phone such as Internet access and communication were noted, and this made learners more confident and technologically savvy. During the interviews, it was observed that learners displayed a sense of accomplishment as they described proficiency in using various phone functions. Thus, exposure to technology in the form of cell phones was well received by the MP group. Positive attitudes towards using technology for learning, together with the reports of high cell phone usage (by both educators and learners), indicate the potential of using cell phones for e-learning or m-learning.

 

What did not work? What were the challenges?: 

Although changes were detected in attitudes towards technology on the part of learners during the project, the findings of this pilot was that there was no significant change in attitudes towards technology and school as a direct result of the technology used by the MP group in the project, given that post-project results for both the comparison and MP groups were generally quite similar. That, is whilst perceptions of technology were enhanced, these perceptions improved across both the MP and comparison group and the extent to which the M4girls project alone contributed to this is unclear.

 

M4Girls: Empowering Female Students Locations

You need to upgrade your Flash Player

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd><p><br> <b><i><blockquote>
  • Lines and paragraphs break automatically.

More information about formatting options