In CourseMill 7.1: FAIL + FAIL = PASS

dave0258
dave0258 Community Member Posts: 102
I produced a module with quiz in Articulate '13 which was hosted on aCourseMill 7.1 LMS. The client noted some odd behavior with how the quiz was working and asked me to take a look. I was able to  get a 100% passing score in the LMS by purposefully failing the quiz, retrying the quiz, and failing a second time. So I built a short quiz to use as a test bed to try a few different settings (like setting the SCORM reporting to incomplete/completed instead of failed/passed, etc.).

To make a long story short, I found no solution, but I did determine that the behavior occurs only when retrying the quiz during the same session OR when returning later and resuming the quiz. I took the test module to the SCORM cloud, and it worked as it should. So in my mind I suspected the LMS, right?

Well, the host of the LMS tried to duplicate my issue and built a test quiz in Articulate QuizMaker. And it worked fine in the LMS. The difference? They used the '09 version. So I produced a functionally identical '09 test quiz based on my '13 test quiz that doesn't work. And the '09 version works.

So what's the difference in publishing to SCORM 1.2 from Articulate '09 to '13? (I've sent this question to the folks at Articulate.) So in my mind I suspect the Quizmaker '13 software right?
But it does work in the SCORM cloud. So my question is whether any CourseMill gurus have seen a similar issue who could shed some light...

Also, if any techies feel like taking a look, I'm happy to share the SCORM package and/or the Articulate Package for testing.

Dave

Comments

  • nhaven1609
    nhaven1609 Community Member Posts: 16
    Wow this one sounds like an interesting one! I would like to look at the SCORM package but it almost sounds like the "lowest passing score" in the SCORM is being met. What where the scores you received each time?

     

    Can you submit your SCORM package at http://cds.trivantis.com/  and I can take a look?
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    Or if you want to post it here, either is fine with me.
  • dave0258
    dave0258 Community Member Posts: 102
    Thanks for offering to take a look. I'm attaching the SCORM package. This version is set up in debug mode, so a second window containing the log will open when you run it. (The folks at Articulate gave me a tip on how to do that.)

    I did find a bunch of "intSCORMError=401" errors in the log, and I'm trying to figure that out -- but frankly, that's not my strong suit. I found in the Rustici "Run Time Reference this definition, though:
    <div id="Undefined Data Model Element2nd3rd4th" class="rt_row_alt"><span class="rt_element">Undefined Data Model Element</span><span class="rt_datatype"> (401) </span><span class="rt_description"> The data model element name passed to GetValue or SetValue is not a valid SCORM data model element.</span></div>
    <div class="rt_row_alt"></div>
    <div class="rt_row_alt">Thanks again, and if you have any questions, let me know.</div>
    <div class="rt_row_alt"></div>
    <div class="rt_row_alt">Dave</div>
    <div class="rt_row_alt"></div>
  • dave0258
    dave0258 Community Member Posts: 102
    A few other notes ...

    The quiz is set to report "Incomplete/Completed"

    The issue seems to show up only when retrying the quiz, or, if you exit the quiz and come back, if you Resume.

    As for scores, I purposely fail the first attempt at 20% (1 of 5). On the retry it doesn't seem to matter whether you answer none or less than 4 correctly. The score is always shown as 100 in CourseMill. And one other note; even though it's set to report as Incomplete/Completed, the status shows up in LMS as Passed.
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    OK, did some testing and some interesting results

    Test 1
    Failed score=0
    Retried test
    Failed score=0
    Clicked finished to close course
    CourseMill status=incomplete score=0
    Relaunched course
    Failed score=0
    Clicked finished to close course
    CourseMill status=incomplete

    Test 2
    Failed score=40
    Retried test
    Failed score=40
    Clicked finished to close course
    CourseMill status=complete score=100

    With the first test getting no completion and score in CourseMill and the second test getting points that could total to a passing course when you add both attempts, it makes me think some manual completion logic is in this course. If this where a Lectora course I would be looking for actions that add up the "points" and possibly setting the score to 100.

    However, the logs that are produced by Articulate don't show the score being set to 100% and that happening. I wonder if there is and logic on the finish button to set score to 100% when some conditions are met.

    Side note: it's very strange that Articulate is encrypting the suspend data.
  • dave0258
    dave0258 Community Member Posts: 102
    Good morning ...

    I replicated your 2 tests with these results:

    Test 1

    Fail w/ score of 0, clicked finished:

    %Completed =10%
    Status = Started
    Score = 0

    Reentered quiz, filed w/ score of 0, clicked finish:

    %Completed =10%
    Status = Started
    Score = 0

    Test 2

    Failed w/ score of 40
    Retried Quiz
    Failed w/ score of 40
    Clicked Finish

    %Completed =100%
    Status = Passed
    Score = 100

    So the only difference is that I'm seeing "started" vs "incomplete" and "passed" instead of "completed." I wonder whether we're on the same version of CourseMill? When I log onto CourseMill, the browser tab briefly shows "7.1." If you're on a different build, I wonder if that makes a difference.

    To your comment about cumulative scores -- when this issue first came up, I thought the same thing, and so did the host of the LMS. But that didn't pan out; by failing twice with a score of 10% each time, that'd still be a failing score of 20%, yet the LMS records 100%.

    That's an interesting thought on the Finish button. But if memory serves, earlier I tested by simply closing the browser instead of the finish button, and I recall the results were the same.

    Now, when I ran the debug log that I attached in the Word document, I tried to be clever in being able to figure things out.

    Since each line is time-stamped, I waited for a new minute to come around before taking the next action. I figured that by doing that, I could isolate each action to a section of the log.

    From my notes:

    1424: entered quiz
    1425: answer Q1 C
    1426: answer Q2 I
    1427: answer Q3 I
    1428: answer Q4 I
    1429: answer Q5 I
    1430: clicked Retry Quiz
    1431: answer Q1 I
    1432: answer Q2 I
    1433: answer Q3 I
    1434: answer Q4 C
    1435: answer Q5 C
    1436: clicked Finish

    So, all the activity on the log related to the Finish button would be time-stamped at 1436.

    I hope this helps -- and thanks again for looking.

    Dave

     
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    Something is defiantly strange with the scoring in thing SCORM package. I put this in the SCORM cloud and had the below test

    Test 3 SCORM cloud
    Take test for a 20% score
    Retake test for a 40% score
    Close the course and SCORM cloud had my score at 60%

    So it looks like the scores from each attempt are a cumulative for the time that the test is being taken.

    Note SCORM cloud does work with the mastery score and CourseMill does.
  • dave0258
    dave0258 Community Member Posts: 102
    Hi Nicholas, I tried to emulate your 3rd test, but each time I ran it in SCORM Cloud, it reported the score of the latest attempt.

     

    Also, when you wrote,"Note SCORM cloud does work with the mastery score and CourseMill does" it almost seems like there's a missing "not" somewhere. Can you verify?

    Also, have you looked at your CourseMill transcript on this? It seems to me that each attemp (interaction) should be divisible by 20, since all the questions are worth 20. I've attached a screen shot showing mine.

    What do you suggest for next steps?

    Dave

     
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    SCORM Cloud does not support the mastery score.

     

    Yeah?? looking at the transcript for my attempts and your, and it is always showing three extra scores, a 50, a 66 and a 100%. It looks like these score are sent a few times from the content. Screenshot is from the CourseMill Full Debug log.

     

    Has this been brought up with Articulate support yet?
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    I also notice in SCORM cloud that the Cloud debugger is seeing additional scores beyond the score that the course is showing on the main screen for the course.

    Example
    The course shows a 40%
    The debugger for cloud shows a 60% for cmi.core.score.raw
  • dave0258
    dave0258 Community Member Posts: 102
    Thanks, Nicholas. I reached out to Articulate with the same message at the same time that I posted on Trivantis, simply because I couldn't figure out who should get the trouble ticket ... you responded first. Articulate sent e-mail yesterday suggesting that I send them the the files as well. I'm putting that together for submission now.

    I'll let you know what they offer.

    Question: does tech support at Trivantis collaborate with Artculate on issues like this?

    Thanks, Dave
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    If you would like help on a conference call with Articulate, I can be available for one. Just submit a Trivantis ticket and say you where working with Nick
  • dave0258
    dave0258 Community Member Posts: 102
    Hello Nick, I just heard back from Articulate Support. I'm copying their e-mail and my reply here. As I'm "the guy in the middle" I'm asking for advice on what to do next. Message follows:

    Dave

    Thanks Angelo. I will pass your question on to Trivantis.

    However, keep in mind that the Articulate 09 version works as expected in the CourseMill LMS. The '13 version does not. That tells me that there's a difference in the published SCORM 1.2 output between the two versions. Because of that, this can't be set aside as solely a CourseMill issue, don't you think?

    I will let you know how Trivantis responds. The tech that I'm working with there is willing to hold a conference call to help figure this out, if need be.

    Best Regards,

    Dave Neuweiler

    561-459-5445

    -----Original Message-----

    From: Angelo Cruz, Articulate Support [mailto:[email protected]]

    Sent: Tuesday, September 13, 2016 14:40

    To: [email protected]

    Subject: RE: Case #00886394: Description in this string: https://community.articulate.com/... [ ref:_00D30Txo._5003310AEKr:ref ]

    Hi Dave,

    Thanks for allowing us time to review the Debug logs. We're seeing quite a few instances of "The data model element in question was not implemented." And you have also mentioned seeing error 401 Undefined Data Model Element (401).

    Our recommendation would be to get back with CourseMill LMS support and ask why they are returning that error so often. Specifically, what data items do they not like us sending? Maybe that the published version is for a SCORM that they don't support?

    If CourseMill can start accepting the data we send or recommend to use a different SCORM version, this will be easier to diagnose.

    You may also want to look into this article for help in working with your LMS:

    https://www.articulate.com/support/storyline-2/troubleshooting-lms-issues

    That should cover the basics including checking with the LMS to see that they are using a supported e-learning specification.

    Let me know if you need anything else.

    Let me know if you need anything else!

    Angelo Cruz

    Articulate Support - http://www.articulate.com/support/

     
  • dave0258
    dave0258 Community Member Posts: 102
    Good morning. I tried a different way, by publishing to AICC instead of SCORM 1.2. Unfortunately, I cannot open the course to test it.

    I've attached screenshots from CourseMill to show what I'm seeing. (On the "access denied" shot, you'll see the course ID is dn-02, while the others are dn-01. I loaded a separate iteration to manually correct the address in the Course Item Details, so this is not a typo.)

    I've also attached the AICC Package.

    Nick, should I be opening a regular support ticket on these issues? Let me know!

    Thanks, Dave
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    I had another SCORM expert take a look at this just to be certain. It looks like the course is recording proper scores and then when you choose to retake the course a second time, the score is set to 100% and then to 0%. Not sure why that would be the case but that is what we are seeing.

    With CourseMill we have the Manage Property "Retain highest score" set to yes by default. This means we will keep the 100% and ignore the 0%. We also support the use of a Mastery Score (which your course uses), so when we get a 100%, your course will get a Passed status.

    Please let me know if this helps.

    Thanks,

    Nicholas Haven

     
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    The issue with the AICC version is that you have an apostrophe int he name of the SCO. Just remove the apostrophe from the name of the sco and the name of the course and republish to AICC.
  • dave0258
    dave0258 Community Member Posts: 102
    Thanks for the update; I've passed it on to Articulate Support.

    Did the other SCORM expert have anything to say about the numerous errors in the Debug Logs?:

    "We're seeing quite a few instances of "The data model element in question was not implemented." And you have also mentioned seeing error 401 Undefined Data Model Element (401)."

    Thanks, and I'll let you how Articulate responds.

    Dave
  • nhaven1609
    nhaven1609 Community Member Posts: 16
    We both agree that we don't know what that is checking for, and to be honest it's not a very detailed error. In most Course authoring tools, this type of error will point to the a item, like the timing information. If we knew what item the course doesn't like i would be able to tell you and Articulate what formats we except for that item and where to find that information in the SCORM ALD documentation.
  • dave0258
    dave0258 Community Member Posts: 102
    Good morning Nicholas. Here's some updated information (I'm just copying an e-mail from Articulate with my reply):

    Thanks, Angelo.

     

    I did try removing the apostrophe on the course title slide; on the LMS publishing information, I could not find any special characters. Anyway, it made no difference.

     

    On AICC, that was one of the first things I tried. But I could not get the course to load in CourseMill. When I opened the course, I kept getting a "server not found" message. On Wednesday, I revisited that approach, and contacted Trivantis about my trouble opening the course. They, too, pointed out the apostrophe as the cause (which it was not, as it turned out).

     

    Although I've published many courses in AICC for clients, never before had I been in the position of having to load one to an LMS. What I did not know was that I had to define the path or launch file name (see below). Once I figured that out, I was able to successfully launch an AICC version, and the quiz performed as it should. My client and I will be testing the entire module today, but I have no reason to suspect there will be any issues. I’ll let you (and Trivantis know about that.)

     

    In the meantime, I’ll forward your question about the “Not implemented error (401)” errors to Trivantis.

     

    Although I have an apparent solution to the issue in AICC, I think it’s in the best interests of both Articulate and Trivantis to sort out the SCORM 1.2 issue. So if either Articulate and/or Trivantis wants to keep working on this, I’m happy to work with both.

     

    Best Regards,

     

    Dave Neuweiler

     

    561-459-5445

     

    -----Original Message-----
    From: Angelo Cruz, Articulate Support [mailto:[email protected]]
    Sent: Thursday, September 15, 2016 17:58
    To: [email protected]
    Subject: RE: Case #00886394: Description in this string: https://community.articulate.com/... [ ref:_00D30Txo._5003310AEKr:ref ]

     

    Hi Dave,

     

    Here're some other things you may want to try or look into regarding this matter:

     

    1) Remove special character in quiz title:

     

    https://www.articulate.com/support/quizmaker-09/the-quiz-title-in-quizmaker-09-should-not-include-special-characters

     

    2) On the 09 file that worked, did you try upgrading the file to '13 format to see if it will work?

     

    3) On the Coursemill FAQ online, AICC is also supported in addition to SCORM 1.2. Have you tried publishing to AICC to see if the issue is specific to SCORM 1.2 of Studio '13?

     

    4) We see a lot of 401 errors according to SCORM run-time reference (http://scorm.com/scorm-explained/technical-scorm/run-time/run-time-reference/) this error is defined as:

     

    Not implemented error (401) The data model element indicated in a call to LMSGetValue or LMSSetValue is valid but was not implemented by this LMS. SCORM 1.2 defines a set of data model elements as being optional for an LMS to implement.

     

    Can your LMS provider check why this error might be coming up?

     

    I'll look forward to your reply.

     

     

    Thanks,

     

    Angelo Cruz

    Articulate Support - http://www.articulate.com/support/

     
  • edt
    edt Community Member Posts: 9
    The cause of the errors could be that Coursemill doesn't support LMSSetValue calls for the cmi.core.score.min and cmi.core.score.max values that are being set within the content. I am not a Coursemill expert, by any means, so that may not be the case. However, I took a look at the content with Nicholas and it seemed that the content would set these values each time a score was set, which caused the 401 errors in the debug of the course.
  • dave0258
    dave0258 Community Member Posts: 102
    Hello Ed, and thanks for joining the conversation. I passed your comment along to Articulate support, and here's their reply:

    Hi Dave,

    It would seem that it may be a limitation of CourseMill that they might not support LMSSetValue calls for the cmi.core.score.min and cmi.core.score.max values.

    Are they able to specify or advise which values does CourseMill prefer? This is so that we can check if there's anything we can modify or settings we can possibly change in your project file to make it work with you LMS?

    Please send us an email once you have an update.

    Thanks,

    Angelo Cruz

    Articulate Support - http://www.articulate.com/support/

    Along with this, I have a question. You're obviously a SCORM expert, but you mention that you're not a CourseMill expert. Is it possible to find and enlist the support of a CourseMill expert? This seems a logical step, don't you think?

    Dave