Question Pooling favours the middle question(s)
trog
Community Member Posts: 33 ✶ Headliner ✶
I did a deep dive on our reporting for some Enterprise wide courses. We have a very large data set to work with and I did some number crunching with excel.
I noticed that when question pooling (randomizing a test section and selecting only to serve 1 question), the randomization favors the middle questions twice as much as the first and last question in the same pool.
Is this a normal expectation of the way the randomization has be implemented? Has any one noticed this as well?
Let me explain further and demonstrate,
In this test results below, Pool (Test section) 7 had 2 questions, where only 1 is served to the learner.
From the reporting data, we get what is expected. Each question is served roughly half the time. Perfect!.
Left side is the responses to each question and the right side is the percentage of the time each question was served. 35,636 got served Q27 while the rest got served Q28
(Note: yes, this test was taken 71,913 times by our learners.)
However, if we have a test section with 3 questions. We get this. Where the middle question (Question 6) is served roughly twice as much as the 1st or 3rd questions in the pool.
Its not isolated, all of the question pools of 3 questions are roughly like this.
Here is the data from my Question pools 4 and 5. The questions are served again on the 25%50%25% basis. I would have thought it should be closer to 33%  33%  33% if it was truly random.
Question pools (Test sections) with 4 questions, where again only 1 is served to the learner. Is split roughly 16%,34%,34%,16%) , rather than 25252525%
Even with a question pool of 9 questions, the middle questions are favored twice as much as the first and last questions. Q8 and Q16 were served half as much, instead of ~11% for each question.
The test was built in Lectora 19. I'm curious if Lectora 21 implements random question serving differently? Have there been improvements?
Cheers
Trev
I noticed that when question pooling (randomizing a test section and selecting only to serve 1 question), the randomization favors the middle questions twice as much as the first and last question in the same pool.
Is this a normal expectation of the way the randomization has be implemented? Has any one noticed this as well?
Let me explain further and demonstrate,
In this test results below, Pool (Test section) 7 had 2 questions, where only 1 is served to the learner.
From the reporting data, we get what is expected. Each question is served roughly half the time. Perfect!.
Left side is the responses to each question and the right side is the percentage of the time each question was served. 35,636 got served Q27 while the rest got served Q28
(Note: yes, this test was taken 71,913 times by our learners.)
However, if we have a test section with 3 questions. We get this. Where the middle question (Question 6) is served roughly twice as much as the 1st or 3rd questions in the pool.
Its not isolated, all of the question pools of 3 questions are roughly like this.
Here is the data from my Question pools 4 and 5. The questions are served again on the 25%50%25% basis. I would have thought it should be closer to 33%  33%  33% if it was truly random.
Question pools (Test sections) with 4 questions, where again only 1 is served to the learner. Is split roughly 16%,34%,34%,16%) , rather than 25252525%
Even with a question pool of 9 questions, the middle questions are favored twice as much as the first and last questions. Q8 and Q16 were served half as much, instead of ~11% for each question.
The test was built in Lectora 19. I'm curious if Lectora 21 implements random question serving differently? Have there been improvements?
Cheers
Trev
Tagged:
Comments

chrystalb21 Community Member, Lectora® Accessibility Group (LAUG) Member, eLearning Brothers® Partner Posts: 27 ✶ Headliner ✶

It's probably a rounding effect. When you pick a random() number between 1 and 3 and round the results, it should be:
1.00  1.49 = Question 1
1.50  2.49 = Question 2
2.50  2.99 = Question 3
This theory of course is quite exactly supported by the statistics above. The effect should get smaller the more questions are in the pool. 
I think I may have a solution; an updated formula. Would love others to weigh in. As I'm not 100% sure on this.
@carlfink , Its not always easy to drill down into the JavaScript, but I'll attempt to below. I love a good challenge.
@timk Your post got me thinking more about noneven distribution and random number generation. So I started investigating further.
I found a discussion on StackOverflow that really suited my understanding. It had sample js for both noneven and even distribution of random number generation.
Specifically,
With this additional information in mind, I investigated Lectora's js files.
In the trivantistitlemgr.js, I found this:
Yay! So I think that's part of the formula anyways. Seems like its using Math.round rather than Math.floor. I can't just change that though. The latter half of the formula has a difference in it too.
I need to find what code parts are calling get_random() function and see what is being passed as (lim).
Further down I noted 2 instances.
One in TTPr.LoadPages = function()
And one in TSPr.LoadPages = function( arr, rand )
If I look at the latter part of the formulas, I noticed this part is not quite the same as my StackOverflow example.
To me the codeMath.round(Math.random()*this.arWork.length  i  1);
looks more like:Math.round(Math.random()*(max  min 1));
If I understand arWork.length and i variables correctly,
rather than the noneven distribution formulaMath.round(Math.random() * (max  min)) + min;
Perhaps there is more that one way of doing it.
If so, if I compare to the even distribution formula, I could infer thatMath.floor(Math.random()*(max  min+ 1))+ min;
in my case could be justMath.floor(Math.random()*(max  min));
Let's toss that on LiveWeave and test these. Both the noneven distribution and what I think is the even distribution on a simulated question pool, random number generation.
0,1 or 2. the max is exclusive while the min is inclusive in the formula.
I ran the code 120 times for a good amount of data and I get the output:
Throwing that data into Excel and tabulating the results.
For the noneven distribution (first formula), we see the same noneven distribution as in Lectora.
The middle number is served twice as much as the first and last.
If we tally up the results using the new formula, we get:
Yay! It appears to be a much better distribution of served random numbers.
So my analysis concludes if I update the Trivantistitlemgr.js file
Updating the first part of the random formula, by replacing Math.round with Math.floor
And removing the 1 in the function under TTPr.LoadPages and TSPr.LoadPages . Not sure which of these functions is used for specifically for questions, but I assume one is for overall tests and one is for Test sections.
Is my analysis sound? I have yet to try this out on actual eLearning courses or on our LMS. I just want to see if my investigation seems on to something. I'd love for ElearningBros to weigh in, perhaps this or something similar could be updated in Lectora 21.
Before I ever use, I would need to do lots more testing, making sure it doesn't break anything or impact a question pool of 2.
Cheers
Trev

chrystalb21 Community Member, Lectora® Accessibility Group (LAUG) Member, eLearning Brothers® Partner Posts: 27 ✶ Headliner ✶
Categories
 35.8K All Categories
 107 ✫ Announcements
 33.1K Lectora®
 31.1K Lectora Discussions
 28.9K Lectora Desktop
 2K Lectora Online
 1.9K Lectora Feature Requests
 71 Lectora User Groups
 36 Lectora Accessibility User Group (LAUG)
 20 ELB Learning Content
 20 ELB Learning Content Discussions
 312 CenarioVR®
 183 CenarioVR Discussions
 129 CenarioVR Feature Requests
 39 Rockstar Learning Platform®
 36 Rockstar Learning Platform Discussions
 108 CourseMill®
 108 CourseMill Discussions
 47 ReviewLink®
 47 ReviewLink Discussions
 5 The Training Arcade®
 5 The Training Arcade Discussions
 932 All Things eLearning
 36 eLearning Development
 545 Learning Management System (LMS) Integration
 331 Web Accessibility
 1.2K ♪ The Green Room
 9 Additional Learning Products