[HN Gopher] Why Aren't SMS Interventions Designed to Boost Colle...
___________________________________________________________________
Why Aren't SMS Interventions Designed to Boost College Success
Working at Scale?
Author : vitabenes
Score : 50 points
Date : 2021-05-10 08:50 UTC (14 hours ago)
(HTM) web link (behavioralscientist.org)
(TXT) w3m dump (behavioralscientist.org)
| infogulch wrote:
| If nagging people to death with notifications doesn't help, what
| does? Maybe goal setting writing exercises:
|
| Using Future Authoring to Improve Student Outcomes -
| https://heqco.ca/wp-content/uploads/2020/03/HEQCO-Formatted_... -
| https://www.selfauthoring.com/research
| dogman144 wrote:
| At scale implies automation, especially guessing at budgets to
| support these programs.
|
| I think schools know there has to be a human at some point in the
| interaction chain.
|
| A frustrating experience I had with this is a school I was at
| (expensive uni) tried to farm out this responsibility to other
| students.
|
| In a mentorship program designed to do 1:1 student:student
| pairings and used "SMS intervention" this program asked students
| to take up to *10* student mentees, with the idea that that the
| real mentor load is pretty low (quick convos).
|
| To me, this felt like an outrageous ask, as if they were trying
| to sub in students paying to be there with the tasks paid
| advisors are responsible for.
|
| This was at grad school, and I think indicated something funding
| challenges behind the scenes. I imagine it's similar/worse at
| high school levels. Maybe not the same exact problem, but it
| makes sense why this specific program linked has trouble.
| saurik wrote:
| This title cut-down to "why aren't text messages designed to
| boost college success working at scale?" was extremely confusing
| until I looked more carefully at the original article and saw it
| says "text message INTERVENTIONS". To save characters--if we are
| ignoring the half-rule to accept the upstream author's title--I
| would suggest changing "text message" to "sms" instead of
| dropping the key noun "interventions".
| vitabenes wrote:
| Changed.
| MauranKilom wrote:
| I still can't actually parse the title. What is "college
| success working at scale"?
|
| Edit: Ok, halfway through the article it became clear: The
| proper reading is
|
| > "Why Aren't SMS Interventions _which are Designed to Boost
| College Success_ Working at Scale? "
|
| and not the way I read it initially:
|
| > "Why don't we design SMS interventions so that they boost
| college success working, at scale?"
|
| Funny enough, this is a case where languages with different
| capitalization for noun vs verb (working, in this case) would
| immediately resolve ambiguities. Well, except for Title Case, I
| guess.
| contravariant wrote:
| It's
|
| ((Why aren't) (SMS Interventions Designed to Boost College
| Success) (Working at Scale))
|
| not
|
| ((Why aren't) (SMS Interventions) (Designed to Boost College
| Success Working at Scale))
| rm445 wrote:
| (Why (aren't (Working (SMS Interventions Designed to Boost
| College Success) (at Scale))))
| tsumnia wrote:
| This isn't specific to the article per say, but something I've
| been discussing with my colleagues revolves around gamification
| and nudge theory in education. There's a lot of pressure on
| gamification in financial trading after the GameStop saga in
| January, and with the pre-existing pressures from loot boxes, I'm
| starting to ask "Why is it okay for education to do it?"
|
| I think nudge theory is a subset of all the elements that make up
| gamification, but still, the idea that something is okay to do
| because "we're doing it to help people" seems a bit odd.
| tomjen3 wrote:
| If you have signed up for college, then I don't think it is
| wrong to get an SMS, similar to what a doctor might send. I
| think the key is "person really wanted this at some point in
| the recent past", which is not the case for gamefication.
| hoppyhoppy2 wrote:
| If you accept that the use of grades is just a way of gamifying
| individual learning to establish player scores and a
| competitive leaderboard, then schools been at it for a long,
| long time.
| SpicyLemonZest wrote:
| When you look closely, a whole ton of things are gamification.
| Even something as simple as a problem set is very gamified, if
| you choose to see it from that angle, since real challenges
| almost never come in the form of neat bundles with a single
| unambiguous solution. I'm honestly not sure whether I think
| that's a useful angle or not, but either way it seems
| inescapable to conclude that there's a category of healthy
| gamification towards good ends.
| crooked-v wrote:
| The tl;dr from the article:
|
| > Implementing a texting campaign with hundreds of thousands of
| students may require organizations to forego advising and have
| texts provide just one-way content or to have large caseloads
| that limit how much engagement advisors can have with individual
| students.
|
| > Another trade-off is that students may have less trust in a
| state agency or national nonprofit sending the messages.
|
| > Finally, state- and national-level campaigns have typically
| sent content that is uniform across all students, whereas
| earlier, local campaigns often personalized information to
| students' college plans or FAFSA (the United States' government
| financial aid application) filing status.
|
| > Even local text campaigns may face obstacles to efficacy that
| similar campaigns did not encounter to the same degree a decade
| ago. [...] As message volume through text increases, we should
| reasonably expect that any one message won't get the same
| attention--and eventually, that people will migrate to other
| communications channels.
| calltrak wrote:
| Integrating SMS messages with your workflow and getting the
| message out there is a no brainer. You can setup something like
| Zapier and https://textgun.io in 5 minutes. Response rates are
| still 5 times better than email.
| michael_vo wrote:
| It does work "at scale" but you need to launch, fail, and iterate
| just like any product.
|
| Georgia State University is a great example of successfully
| implementing the goal of "college success" and continuing to
| innovate. They started by solving the problem of students
| accepting their offer letter, but never choosing classes during
| that crucial first summer.
|
| First tool they built used a combination of AI chat box and
| counsellors to answer common questions, help students fill out
| forms, and/or get financial aid. At each step of product
| development they asked "what's next?". For example choosing a
| degree is "hard" for first generation college students. So they
| built salary, and career earnings during the UX where you pick
| your courses.
|
| Another non-SMS solution they provide is automatically depositing
| money into student's accounts who are low and struggling. This
| especially helps 4th year students who may have burnt through
| their financial aid.
|
| https://www.youtube.com/watch?v=Rh20W2uXHCc&ab_channel=BillG...
| impendia wrote:
| I'm a professor. My university bought a big piece of enterprise
| software, designed precisely to track and boost student success
| in various ways.
|
| The software sent me an email asking me to do something,
| ostensibly from someone in our advising center, signed by my
| assistant chair instead, but obviously corporate-sounding and
| autogenerated. It reeked of fakery.
|
| Is this sort of thing supposed to accomplish something?
| fallingknife wrote:
| Yes. What it accomplishes is to abuse the trust you have built
| with your students through genuine interaction to spread their
| corporate (ok university. Same shit) propaganda by making it
| seem like it comes from you. Most people can smell that from a
| mile away so you should absolutely not do it if you care about
| the respect of your students.
| Loughla wrote:
| That tells me you have a team of people who have not done their
| jobs, or let too many decisions be made by committee.
|
| We have a large piece of software at our institution that
| communicates automatically with students, staff, and faculty
| based on various triggers (grades, dates, etc). The hardest
| part of that process was writing the messages. It took 1
| semester to get the system operational, and two years to dial
| in the messaging.
|
| The worst thing you can do is create a "specific" message for
| an individual with generic language targeted at groups. Which,
| subsequently, it seems is what most institutions do. No idea
| why. It's easier, I suppose?
| toast0 wrote:
| > The software sent me an email asking me to do something,
| ostensibly from someone in our advising center, signed by my
| assistant chair instead, but obviously corporate-sounding and
| autogenerated. It reeked of fakery.
|
| This is like the definition of phishing.
| ineedasername wrote:
| It depends on how it's implemented. These systems _can_ work,
| but they require very tight coordination & buy-in from faculty
| as early as the procurement stage. Schools that just buy &
| force them on faculty don't achieve much.
|
| The types of messages faculty get like that are usually a
| check-in where they're asked to give a status report for each
| student. This can be a quick good/ok/bad type of thing, or it
| can be something like a brief survey. My personal opinion is
| that the first option is best: These systems should put as low
| of a burden on faculty as possible. I really minimalist
| implementation that is almost as good is to ask faculty to
| _only_ flag students in trouble.
|
| What this _should_ do, if the college has properly implemented
| it, is trigger the advisor to reach out to students struggling
| to determine 1) Why and then 2) What can be done to help:
| tutoring, assessment of whether the course is really required,
| determination if the course can be withdrawn from if not
| required or if the student is beyond the point of recover, or
| should have taken a pre-requisite or lower-level course in the
| subject first, etc.
|
| If your college has not communicated this sort of thing to you,
| they are doing it wrong. I would still encourage you to do what
| is asked unless it is a very large burden. If that's the case,
| reach out to the advising unit overseeing these initiatives to
| understand how they're using it. But if it's not too much of a
| burden, even if they've done a poor job communicating with
| faculty, it may trigger a meaningful intervention with the
| student. Even if it doesn't, it will still allow collection of
| data that can be analyzed to determine trends, the significance
| of whether these activities by faculty correlate with student
| success, etc. And if they do, _that_ will then be used to feed
| back into the advising process that these things are actionable
| to the benefit of students.
|
| These systems _can_ work. In my current job, after implementing
| such a system we saw first-year drop outs decrease by about 6%.
| myfavoritedog wrote:
| Yes, the people who sold the software to the university made a
| ton of money.
| ineedasername wrote:
| I work in the industry. There are a couple of reasons they don't
| work, at least as currently used.
|
| 1) Over saturation/overused: Especially as High Schools begin
| using these, students get too many, too frequently, and it just
| becomes a sort of background noise. Back when students still had
| email as their prime mode of digital communication, we had the
| exact same issue. Studies on their effectiveness were based on
| relatively short time spans where nudges were new & novel: The
| novelty has worn off.
|
| 2) A simple nudge can't help a complex problem. A college may
| nudge "You haven't registered yet! Don't forget to register for
| the next semester". That's an extremely frustrating message to
| receive when you're poor and can't pay next semester's bill, or
| some other roadblock.
|
| 3) It has been used to justify not scaling human resources for
| students, so any benefit is offset by lesser real human contact
| with advisors. People complain about administrative bloat at
| universities, but that doesn't apply to advising: I've seen
| caseloads rise from ~300/advisor to ~500/advisor. This is
| especially problematic for overcoming #2 above: Advisors can
| often help with complex issues that student's believe are
| insurmountable.
|
| My observations on how to do it better:
|
| Only use nudges for low-stakes issues, and even then choose
| carefully so students don't become numb to them. High-stakes
| issues need human outreach. With limited bandwidth by advisors to
| do this work, it needs triage: Even if it's high stakes, don't
| waste time reaching out personally to low-risk students.
| Determining risk requires a two-pronged approach:
|
| 1) A statistical model that uses historical data to identify
| students at risk of poor performance or attrition. A competent
| data analyst can build a reasonable model without having to go
| too far into advanced data mining. "Deep Learning" tends to be
| fairly useless here: There's not enough data. There's also a few
| good vendors out there that will perform more advanced risk
| modeling, partitioning a school's population into a dozen or more
| groups and building models for each of them. These aren't cheap,
| but I've seen it pay off when correctly paired with one-to-one
| human outreach initiatives. In a single year I saw it reduce
| attrition by 1%, not to mention academic performance increases,
| and that more than paid for the cost. I've also seen schools pay
| the invoice and expect things to magically change without
| aligning their resources to the risk analysis output and waste a
| lot of money.
|
| 2) A rule-base system of identifying students at risk. Things
| like a late payment; a good student getting their first D or F in
| a class; delaying registration for the next semester; requesting
| transcripts to be sent to another school; and many other factors.
|
| Using the high risk population identified in #1 and #2, _that_ is
| where you focus your one-to-one human outreach efforts.
| jrm4 wrote:
| Higher-ed Instructor Here.
|
| Because, and I am not exaggerating in the slightest, this is the
| _worst_ idea I have ever seen directly related to my career. This
| concept is an idea that I am fighting all the time.
|
| Teachers and students alike around me sort of blindly "get used
| to" these notifications flying around you and even begin to
| expect them even when they don't work.
|
| Essentially, they become the equivalent of spam in your email
| that you ignore, except they're harder to ignore and people still
| have this wildly misguided faith that they can think for you.
|
| As a first measure -- I'd love to _remove_ all "notification"
| type services from higher-ed; after a bit of a learning curve, as
| in -- yes, you must visit this website or call this person or
| read the syllabus to get things done.
|
| You develop your own system of remembering things, maybe it's
| paper, maybe you email yourself, whatever works _for you_
| xwdv wrote:
| SMS used to be a very effective channel for getting conversions
| from people. But marketing channels are limited resources, they
| are most effective early in their discovery when organizations
| begin using it, but once their novelty is sucked dry the
| effectiveness tapers off until it's no more effective than a
| flashing banner ad.
| thrower123 wrote:
| Yikes, the whole concept of this sounds horrifying to me. That
| text message vibration and notification ping engenders an
| immediate stress reaction.
| orangeoxidation wrote:
| Do they not? "At scale" is somehow often synonymous with "with
| less effort and care per instance", but that doesn't have to be
| the case. Things work worse if you scale only one part of the
| equation (the "customers"), but do not scale the other side
| appropriately (the "service provider"). That should be obvious.
|
| > Text campaigns are most likely to be effective when they:
| Are sent by a school and even staff member who students know and
| trust Add meaningful and novel value above and beyond
| existing outreach and support efforts like giving students real-
| time information on the status of their financial aid
| applications Provide content that is personalized to the
| student and where they are in the college application and
| financial aid process
|
| Having a new org sending out mass messages to all students is not
| scaling The described. It is something different.
|
| Scaling it would be convincing more school staff and schools to
| make a similar effort.
|
| The positive effect was not caused by "getting a message", it was
| caused by "getting personal advice from someone you know". They
| scaled the wrong thing.
| dkarl wrote:
| Things also "work" if you are able to scale down a cost
| (employment of human labor such as teachers and counselors)
| while maintaining or scaling up a source of revenue (student
| enrollment.)
|
| I worked for a company that offered this service (among other
| data-y things.) It employed lots of bright-eyed young people
| who were extremely happy they had found a tech job where they
| could do good, helping struggling students succeed at college.
| It was a dream job. However, the information we heard trickling
| in from the customers (public universities and community
| colleges) was pretty demotivational. They ones that were
| shrinking were panicked about their future existence, and the
| ones that were growing were basically like, yeah, hiring a
| proportional number of front-line staff to interact with
| students like in the past would just be silly. Nobody is going
| to do that. Run along, idealistic tech people, save these poor
| kids with analytics and text messages, because we're not paying
| to do it the old-fashioned way.
| ineedasername wrote:
| _Things also "work" if you are able to scale down a cost
| (employment of human labor such as teachers and counselors)
| while maintaining or scaling up a source of revenue (student
| enrollment.)_
|
| This is an unfortunate truth. Schools grow, don't scale human
| labor to accommodate additional students, overall student
| performance & dropouts increase, and their answer is to
| recruit more students in a downward spiral. Never mind the
| fact that it costs more money to do this than simply hiring 2
| or 3 more advisors to do real human outreach. I saw this
| cycle over the course of 4 or 5 years where I work. After
| providing hard analysis of this pattern each year, I, working
| with colleagues on the academic side of things, managed to
| push for those advising resources and the results have more
| than paid for it.
|
| The problem is that many of the idealistic tech people are
| only looking at the problem from the tech point of view. I
| have resisted efforts over the years to assimilate me into a
| central IT division because the viewpoint I get through being
| embedded in an operational division strongly informs my
| analytical efforts. I see the business cycle, I talk to
| advisors, I construct & analyze surveys that mix quantitative
| _and_ qualitative factors-- there is no detail of the student
| experience that is too small for me to ignore. It hammers
| home the point that analysis & tech can only help focus the
| efforts of human labor, not replace it.
| newnamenewface wrote:
| This really gets at the heart of why I have very little
| allegiance and essentially no desire to donate to my alma
| mater: Why would I care about or donate to a business that
| really doesn't care about me individually besides $$$ signs?
| Sure, I patronized them for four years of my life, but I've
| done the same with In & Out for longer and you don't see me
| sending them a check for nothing. Much like In & Out, the
| positive memories of my alma mater were more so about the
| people I was with than the location or business proposition
| itself. Ok, I'm ending this metaphor before it becomes too
| contrived...
| toomuchtodo wrote:
| Better to donate to Khan Academy. Can even enjoy an In N
| Out burger while doing so.
| ineedasername wrote:
| Khan academy is absolutely fantastic. However, it's only
| a step above giving students free textbooks or other
| self-learning resources: It mostly work for students that
| are already self-motivating. This tends to be a minority
| of students, and most require some degree of a human
| support network to succeed.
| hattmall wrote:
| Realistically you donate so that the school has a bigger
| budget and more opportunity to gain prestige making your
| diploma more valuable. That's really only relevant to a
| small subset of the population and rapidly shrinking. Four
| decades ago degrees had meaning but now they are largely
| participation trophies that come at great expense.
| Spooky23 wrote:
| Some problems can be improved with a simple contact point.
|
| I have a colleague who works in this space in a research
| capacity. They were looking at use of different systems to
| measure behavior and identify behaviors that are correlated
| with poor performance.
|
| In one scenario, they surveyed a cohort who was leaving and
| entering the dorm after 1AM, resulting in poor AM class
| attendance. 40% of the responses were essentially: "I went to
| meet my [girl/boy]friend and have sex."
|
| Getting a text message to tell that kid to get his beauty rest
| isn't going to work. Just treat people like adults, imo.
| ineedasername wrote:
| _Some problems can be improved with a simple contact point_
|
| Absolutely. Most schools don't employ enough advisors to have
| sit-down appointments with every student every semester.
| However, I did see a school implement a policy where 1st
| semester students-- a higher risk population-- were required
| to have a sit down like that before they were allowed to
| register for the next semester. That single effort resulted
| in a 15% decrease in 1st-year dropouts.
| Spooky23 wrote:
| Makes sense.
|
| I went to a big public university where they had two
| freshman tracks, a "college within the university" where,
| if you knew to sign up, you'd be in a cohort of students to
| meet some general education requirements and would get
| access to faculty motivated to help you. The problem was
| that the kids who needed the least help tended to discover
| this program.
|
| The "normal" path was being dumped into the University
| bureaucracy blind, and you had to learn to sink or swim. It
| was tough -- the registrar in those days was an experience
| like DMV. I'm pretty sure the system was devised as a
| funnel to disqualify people for popular majors like
| Business, Computer Science, etc.
| ineedasername wrote:
| _the kids who needed the least help tended to discover
| this program_
|
| Heck yeah did you hit the nail on the head. I've analyzed
| the data over a period of years for which students
| utilize tutoring services: They very much tend to be
| students that were already doing alright _before_ they
| started getting tutoring.
|
| Conscientious self-motivated students are precisely the
| ones most likely to seek out help on their own. As a
| result, for students that go on academic probation we now
| _require_ them to attend a certain amount of tutoring
| hours each semester, and the software we use to manage
| advising caseloads allows faculty to raise a flag for
| students performing poorly so that advisors can pro-
| actively reach out and direct them to tutoring resources.
|
| This has led to a huge increase in tutoring resource
| utilization, but unfortunately they are now at the limits
| of their capacity. Our next hurdle is to analyze the
| academic outcomes of students using those resources,
| which will _hopefully_ show increased performance &
| allow us to petition for more support services.
|
| Fortunately our leadership is not completely short-
| sighted on these issues, but as obvious as "more one-on-
| one advising & support" would seem, they (not too
| unreasonably) still demand very concrete evidence to
| justify additional expenditures. However it would be nice
| if they viewed customer service of this sort as the
| profit center it is, rather than a cost center. The
| problem is that things like the benefits of quality
| tutoring of freshmen may not yield tangible results for a
| few years, so things move slowly.
| splithalf wrote:
| Excellent point. It's a problem with operational definitions
| which plagues a lot of research.
|
| But "scale" in this context implies economic efficiency. You
| can't give every kid a full time counselor. You can write a
| script to send sms messages.
|
| One question for the tech community, to what extent can
| advances in ai and other technology impact this question of
| scale/efficiency. If the student doesn't know his counselor is
| a bot, maybe the nudges become more impactful.
___________________________________________________________________
(page generated 2021-05-10 23:02 UTC)