I have launched or participated in more than 40 educational projects and products
A new way to measure overconfidence bias, published paper, 2013
Kolesnikova, D., & Belianin, A. (2013). Prognosis Calibration in Binary Choice Tasks. Psychology. Journal of Higher School of Economics, 10(4), 42-66.
Abstract This works focuses on the problem of confidence in decisions made. Existing studies of prognosis calibration (defined as the ratio between the correct response frequency and the subjective evaluation of response correctness) have revealed that people tend to overestimate the probability of correct responses ("overconfidence effect"). The authors hypothesized that subjective estimates of confidence in decisions made and the accuracy of prognosis calibration depend on the method used to measure confidence.
An experimental study was performed in a sample of school students (N=50) and university students (N=36). The participants answered common knowledge questions and evaluated the degree of confidence in the correctness of their answers on verbal scales, using either a standard scale or a duplex scale (rating their confidence in both chosen and rejected answer options). Finally, the participants made bets that the chosen alternatives would be correct.
The results indicated that the "overconfidence effect" was weaker in the duplex scale condition, confirming the hypothesis. We explain this finding by the notion of residual uncertainty that remains when two answer options are compared, but not in the standard scale setting, when the respondents' attention is concentrated only on the chosen answer option. The association between the level of confidence and prognosis quality had an inverse linear shape in the standard scale setting and a parabolic shape in a duplex scale setting. Finally, we found that in cases when the decisions were rational and the confidence ratings were consistent, bets predicted prognosis quality more reliably than verbal estimates did.
Details This work started at Teachers College, Columbia University, New York in 2012 as research for my Master thesis and was finished in collaboration with Prof. Alexis Belianin at National Research University Higher School of Economics, Moscow in 2013.
Comparison of ipsative and Likert-type scales
Term paper 'The Theoretical Framework for Construction and Comparison of Interest Surveys based on Normative and Ipsative Scales' 2019 – present
Description The work on this paper consisted of the following stages
Based on Social-Cognitive Career Theory the framework for vocational interests questionnaire in Data Science was constructed. Popular occupations in Data Science were decomposed into 4 dimensions: math, programming, analytical thinking, and leadership. Potential career choice was modelled as a combination of self-efficacy, outcome expectations, and occupational interests.
44 statements were developed based on the framework. These statements were prepared for both Likert-type and ipsative questionnaire.
Based on three cognitive laboratories, 40 statements were chosen for the next stage, numerous statements were re-written.
Questionnaires were piloted. 84 respondents participated in the pilot study. Classical Test Theory was used to assess the quality of the items formulated for Likert-type questionnaire. Based on this analysis, statements were re-written once again.
New versions of the questionnaires were tested in 5 cognitive laboratories. They were re-written.
Randomised design was developed for the next round of surveys exposure. The last step will be to gather new dataset for both questionnaires. The analysis will include IRT for item quality assessment for the Likert-type survey and transformed data for the ipsative survey. Then SEM will be used to test whether the data fits the theoretical framework. Finally, prediction models will be built to test whether the questionnaire could be useful for prediction of one's vocational interests.
Supervisors Maxim Skryabin Denis Federyakin
Further details This work has been developed during my Master's program in Educational Measurement at National Research University Higher School of Economics, Moscow
Short draft, version Oct, 2020
Assessment of student progress at the online school, pilot research
Pilot research on the knowledge progress at SkillFactory, for students at Data Science and Analytics programs, 2020
Description It was necessary for the online school to understand whether its students develop during their studies. For the pilot study two courses were chosen: 12-month program on Data Science and 6-month program on Analytics. The work on this project consisted of the following steps:
Pedagogical specifications for both courses were revised
Based on the specifications, two knowledge tests were designed in collaboration with domain experts (15 items in the Analytics test, 25 items in the Data Science test)
47 and 255 respondents participated (Analytics and Data Science correspondingly), they represented 16 and 29 cohorts who had started studying at different time
CTT and IRT were used for the analysis of Analytics and Data Science tests respectively
It was shown that, at the Data Science program, students who started to learn earlier showed significantly higher knowledge level than more recent cohorts. This difference was not so significant for the students of the Analytics course. Some knowledge gaps were discovered. The tests were revised based on the item quality analysis
Details I initiated this research while working in SkillFactory as a product manager at Quality Lab. The goal of our team was to raise the quality of our educational content
Game-based assessment of 21st century skills
Four online pilot games created in two weeks, Feb 2018
Description The idea was to create games for 21st century skills assessment. All games were based on the famous experiments from behavioral economics or cognitive science. As a result 4 online games were created from scratch just in two weeks with a team of 4 people. Games were piloted with a 200+ children in cooperation with an educational platform InternetUrok.
Four competencies that games were aimed to measure: – Ability to learn (the game was based on the theory of concept formation; Voiklis, J., & Corter, J. E. (2012). Conventional wisdom: Negotiating conventions of reference enhances category learning. Cognitive Science, 36(4), 607-634) – Resource management (the game was based on the ability of respondents to act rationally; Shin, J., & Ariely, D. (2004). Keeping Doors Open: The Effect of Unavailability on Incentives to Keep Options Viable. Management Science, 50(5), 575-586) – Ability to concentrate (the game was based on the classical n-back task; https://en.wikipedia.org/wiki/N-back) – Ability to manage risks (this game was more about personal preferences than abilities; it was based on the Tanaka, T., Camerer, C. F., & Nguyen, Q. (2010). Risk and time preferences: Linking experimental and household survey data from Vietnam. American economic review, 100(1), 557-71)
This project doesn't have any scientific value, as I understand right now. However, it is because of this initiative that I decided to dive deep into psychometrics.
Further details Try a prototype: http://184.108.40.206/#/login Login: testphd Password: passphd20
Experimental research on public goods
Research assistantship at the Lab for Experimental and Behavioral Economics, HSE; 2010, 2012-2013
Description I participated in data collection for public goods experiments in Perm, Tomsk & Moscow. In particular we gathered data to explore the phenomenon of spite. The spite could arise when respondents saw that someone contributed more points to the public good than they did. This effect was found to be especially strong in Middle East, Russia and Eastern Europe (Herrmann, Gachter, Thoni, 2009).
The goal of the experiments conducted by our lab was to explore true motives of the people who showed such anti-social behaviour. It was found that such motives are numerous: availability of the option to punish, competitiveness, preemption, etc.
The conclusions were based on 300 paid participants. 128 from Moscow, 76 from Perm, 96 from Tomsk. The experiment was realised in the software z-tree.
Supervisor Prof. Alexis Belianin
Presentation with the results
I am a founder of the startup called MyStory ('Obraz jizni' in Russian), 2014-2019
The initial goal was to create an opportunity for every child to explore the world around. 3.5 years ago we started with online-meetings for teenagers living in distant orphanages, today we work with both young people from regional orphanages and high-school students from all over Russia and CIS. By working with two groups of participants we are able to be financially stable. Currently our focus is on offline activities, for example city camps.
10 products in career counselling (online meetings, trainings, school courses, camps, etc.)
More than 18k participants in online course
More than 500 hours of events
More than 400 unique participants from all over Russia in summer camps
17 regions including Armenia and Ukraine
Won the competition for funding from federal government, 2018; from Moscow government, 201
Our website (RUS)
Python course in SkillFactory
SkillFactory (Data Science School, more than 4,000 students, 30 online-courses). Launched Quality Lab - the department for quality control. As a product-manager increased gross profit from 30% to 55%, quality from 23% to 60% (NPS). Launched 'New products' department (free experimental products).
Economics based on games course
The idea was to show that Economics as a science doesn't study only finance and business but analyses human behaviour under different settings: from shopping to dating. This project is currently on hold, though I conduct some rare events on non-standard economics and teach classes in the school on weekly basis.
More than 800 participants overall
From 7 to 60 years old
20 events in 1 year
2 year-courses for schools are developed and conducted
Teaching at the school
Taught Economics for school children in High School, 2015-2017 School №1533, Taught Behavioral Economics for school children in Middle School, School Intellectual, 2014 Designed and taught 10+ workshops on Statistics and Economics, 7-12 year old children, Polytechnic Museum, Moscow 2015-2018 (Independant)
Board game development
Developed a board game about communication for top-managers, Masterskaya O. Zamyshlyaeva
Teaching related to career consulting
Workshops, webinars, online and offline courses, 2014 - 2019
Developed free online course about MyStory approach to career counselling and published it at Stepik.org, 2019
Conducted more than 30 talks and workshops [RUS] for school teachers on the methods of career counselling and support of young children, MyStory, 2014-2019
In partnership with a leading online Russian school launched 2 online course for teachers about career counseling that attracted more than 18K participants and certified more than 10k teachers (one course in now archived, another is available here [RUS]), 2017, 2019
Developed a course on career choice for 10th grade, 2017-2018, Lyceum of HSE (MyStory
Online course at Stepik.org
For teachers about MyStory approach to career counselling, 2019
Developed free online course about MyStory approach to career counselling and published it at Stepik.org. It includes 6 chapters on how to organise engaging meetings for young people and professionals from various occupational spheres at school.
2021. NRU Higher School of Economics, MA Educational Measurement
2013. Columbia University, Teachers College, MA Cognitive Studies in Education
2011. University of London, International program, BSc Economics, First-Class Honours Degree
Sep 2020 – now. R&D Senior Product Manager, uchi.ru (Online Platform for primary school students; more than 8.5 mln users). The task is to launch R&D department.
March 2019 – May 2020. Head of Student Experience, SkillFactory (Data Science School, more than 4,000 students, 30 online-courses). Launched Quality Lab - the department for quality control. As a product-manager increased gross profit from 30% to 55%, quality from 23% to 60% (NPS). Launched 'New products' department (free experimental products)
2014 – 2019. Founder, NGO MyStory (21st Century Skills to the Youth), obrazobraz.ru; Launched more than 10 products on career counselling, including online course for teachers (more than 18k registered, 10k finished; camps for young people - more than 400 participants from all over Russia, etc); Won the competition for funding from federal government, 2018; from Moscow government, 2017
RESEARCH AND TEACHING EXPERIENCE
June – Aug 2020. SkillFactory. Introduced metrics of content quality, student knowledge level, retention.
Oct 2019 – now. Developed an interest inventory for IT newcomers. The goal is to compare an ipsative and Likert scales functioning.
2012 – 2016. Taught Economics and Behavioral Economics to kids, high school students, university students, economics teachers; designed more than 50 educational courses and events.
2010 – 2014. Research assistant, International Laboratory for Experimental and Behavioural Economics
2012. Research assistant. Columbia Uni, Teachers College; James Corter's Cognitive Science Lab
2020. HSE Alumni Awards
2018. A member of YPO (Young Presidents Organisation) Impact program
2018. Winner, second prize, Innovate for change, Berlin
2017. Included in the list of innovators in education in Russia
2014. Absolute winner, Competition of Innovations in Education
2010. Letter of Commendation from University of London,
2008, 2009. Potanin Fund scholarship student (excellent marks and teamwork skills)
2009. Oxford Fund scholarship student (excellent marks and activities)