by Ruchit Nagar
We are currently conducting a Randomized Controlled Trial to rigorously evaluate the effectiveness of the Khushi Baby system, which serves as a multicomponent intervention. Across 81 Subcenters, mothers coming to the 300+ serviced camps will receive a Khushi Baby necklace upon registration and a series of voice call reminders in their local dialect. The nurses in these subcenters will be using the Khushi Baby app, which includes features like due list reminders, quick patient record identification with NFC, high risk alert messages, camp summaries, motivational messaging, and progress tracking. Nurses are still compelled by government protocols to fill out paper registers and Mother health cards (MAMTA cards), but they can do so quickly with the assistance of our application. Health administrators will receive access to the features of our Dashboard: action plan builder, attendance maps, district snapshot, find patients, voice call reminder tools, and downloadable report formats. The Data Entry Operators too have a stake in using the report formats for the 40 Primary Health Centers in which the intervention is underway for their reporting to the State Pregnant Women and Child Health Tracking System, the electronic health registry known as PCTS.
Naturally the question arises, what is the best way to evaluate the effectiveness of our intervention. We started off with the metric of improving health behaviors, specifically on-time infant immunizations. We also hoped that our intervention could improve antenatal care adherence (coming to four checkups before birth) and hospital birth rates (as opposed to home births) to a statistically significant and practically meaningful degree (>10% improvement from baseline). Each of these health behavior outcomes is independently tied to impacts on maternal and infant mortality and morbidity.
But can we actually measure a difference?
In control camps there is no real gold standard for evaluating whether a mother finished all four antenatal care visits, had a hospital birth, or if her child received all his/her immunizations. Paper records (either carried by the mother) or in the RCH register do not concord well with the state e-health registry. We have heard about data manipulation from our field experiences with the nurses themselves. Ultimately, oral history is taken from mothers who have low literacy and health awareness, often well after the health engagements took place.
The issue is data.
What if our system shows a decrease in health behavior outcomes when compared to the control group? That does not indicate that our system has indeed failed, as the result could be a consequence of the discrepancy of data quality.
Therefore our framework for evaluation shifts to the following
- Can we trust KB data, more so than the status quo system?
- Does the KB system and the data collected turn into actionable engagement, the kind of which can lead to health behavior change?
With this in mind, our primary metrics focus on data:
- Data quality – how well does the backend data (KB vs PCTS) concord to the mother’s copy (neckalce vs. MAMTA card)
- Data timeliness – how long does it take for the data to reach the backend from the field on average
- Data actionability – how many specific engagements were triggered as a result of data presented on the frontend or backend per month in the KB system
- Data retention – how many mothers and children retained their KB necklace as compared to the paper card
But before we can get to the metrics, there is a bigger fundamental question to be answered: will the KB system actually work?
As our intervention is being rolled out, we are grappling with the everyday field realities that challenge this basic assumption.
The cornerstone of our system is the Auxiliary Nurse Midwife (ANM). She delivers care to mothers and infants and records the engagements that took place. She is a champion who we hope to empower with our system. But there are barriers for her to collect and record our most essential currency: health data.
First, some ANMs have basic challenges traveling to the health camp to begin with. If the ANM cannot show up to the camp, then there of course is no service to be provided or data to be gathered. Furthermore, the health administrator has no idea, in real-time, which nurses are actually attending, and which ANMs have falsely created health records to cover up their absence – until maybe now. With the KB dashboard, health administrators, should they engage with it, can see in real-time which ANMs have checked in, and whether they have checked in at the correct location for their scheduled camp. Responding to truancy however may not be as simple as simply observing it.
Second, some ANMs are pressed for time at the camp setting itself. When ANMs are pressed for time, they are less likely to conduct required tests, collect data, and record data. ANMs may be pressed for time due to high patient attendance or because of the time lost to commuting to these rural village based settings. Their time may be saved with the support of paraworkers at the camp setting, but the ANM sees variability in the help that she receives from camp to camp.
Third, the incentive structure for the ANM situates an unrealistic expectation to improve health behavior outcomes, without a mechanism for accountability or the tools to make such incentives attainable. The ANM is only able to persuade health camp adherence or follow-up care adherence within the communities to a certain extent. Above this extent, the supporting community players such as the ASHA and community leaders, along with the incentives to the beneficiaries themselves must fill the gap. If improving health behavior metrics are not in the ANMs control, and if she is still expected to increase these values, she does so by hand – literally manipulating data. This data manipulation is not only evidenced by our ANM interviews, but is explicitly captured in the difference in linelisting data with aggregated counts that make it to the state EHR. In fact some ANMs have expressed concern that they will no longer be able to manipulate the data on the KB system.
The ANM also rightly may believe that the data has no value to the managers to which she reports. The FORM 6 and FORM 9 reports that she has to fill on a monthly basis therefore a are simply a burden – to be filled out quickly and without consequence. Who is actually looking at the data and using it for follow-up? Even the mother’s paper card – which may be lost, forgotten, and is often not fully filled out – does not have the reputation of being useful to any other health provider, even though that may have been its intended design to begin with. If no one is using the data, why spend time to collect it?
In some cases the data they collect would be faulty anyways, and therefore not valuable to anyone else. For example, the BP cuff or scale may be miscalibrated. And while some ANMs have glucometers, they are still expected to purchase the testing paper for 35 INR for later reimbursements. ANMs don’t purchase the strips because in practice they know they won’t get reimbursed.
And the government mandate to fill out the RCH register and MAMTA card at the point of care rarely holds. ANMs have their own SOPs. They can be seen filling some data in their personal diaries, and later filling the RCH register at the Subcenter – in some cases inventing information as they go. MAMTA cards are held by the ANM instead of distributed to the mother, so that the ANM can “ensure” that the the mother or child receives all the due checkups.
The data we have collected so far reflects data manipulation in the field. 100% of the several hundred mothers we have enrolled show no signs of either albumin or sugar in their urine. For over 150 infants, there are zero reports of diahrrea or pneumonia in the last two weeks. Over 80% of BP measurements are multiples of 10 and in the normal range. The health administrators are fully aware of this data manipulation.
“110/70 is our ANM’s favorite number” – Udaipur District Reproductive Child Health Officer
Our COO, Md. Shahnawaz, shared a poignant example of a nurse who had to travel with her husband to get dropped off at the camp. The nurse was in a rush at the camp because her husband and her ride, a school teacher, was losing his daily wage and was eager to return back from the camp. Shahnawaz offered to give a ride back to this nurse, freeing her husband to get back to school, and allowing the ANM to take her time. When he asked her earlier to take things according to the standard process, out of frustration she responded:
“What’s the point, these patients are going to die anyways. They don’t know anything about their health.”
Perhaps this cultural barrier of apathy and fatalism may indeed be the most damning of all. If no member of the health value chain from the ANM to the health administrator, believes that they can truly move the needle, the status quo will not change. ANMs out of a moral sense of duty may conduct their camps and inform mothers to go for follow-up care at a referral facility, but there will be no sense of week to week improvement. Health administrators will continue to provide “supportive supervision” to low performing blocks, but ultimately fall back on the crutch of a broken and data-poor accountability mechanism.
So what can we do about it?
We have to truly empower the ANM and the health administrator alike.
We have to show them that the services they provide, the data they collect, and the actions they take do translate into tangible health outcomes.
We have to reduce their tedious work, and make their work more efficacious.
And we have to support them, and show exactly how we can integrate this system into their existing work plans.
The KB platform (the “what”), and more importantly the way it is implemented (the “how”), can achieve these objectives.
Here’s what we are doing to make it happen:
- For 81 nurses we have a team of 16 monitors and 4 core staff to provide support; each day before the camp, each monitor will work with 3-4 ANMs to backfill retrospective data on our platform (attaching labels to the necklaces of existing patients). This achieves two objectives: providing more hours for the ANM to get comfortable with the platform and second, reducing the load of patients the ANM would need to enter on the camp day itself by 80% to only those who are first time camp-attenders. In other words if 15 patients come, 12 will be given a necklace as a returning patient, with minimal app-entry required and only 3 will need to be fully enrolled. This should allow the nurse to spend more time collecting and recording the data for that antenatal care checkup. This backfilling process will take place over the first month as ANMs need to be ramped up and independent, and so that all preexisting data can be backfilled.
- We are providing due lists and high risk alerts to the ANMs. They can use these lists to directly communicate with the ASHA to mobilize mothers to camps or for follow-up. This data-driven engagement can directly translate into the health metrics they are incented to increase. If they correctly fill in the mother’s phone number and health data, our automated voice call system will further assist the ANM and ASHA by providing timely educational messaging and reminders via dialect-specific voice calls. The alerts also reinforce the clinical significance that the tests convey, so that the ANM understands the value in collecting the information appropriately.
- We provide a mechanism for the ANM to report whether she had faulty equipment, no equipment, lack of training, or faced resistance from the mother when attempting to conduct a test. We provide denial reasons also for medications and vaccines. In this regard the data that the ANM records directly feeds back into action steps which can empower her to administer these tests and therapeutics in the future.
- We allow the ANM to track her progress on a monthly and all-time basis, and we look to integrate metrics such as “referrals completed” along with health metrics for pregnancy, child birth, and infant care.
- We provide remote support
- We are engaging the Data Entry Operators and LHVs who till date have had very little control over the data quality. Data Entry Operators now have a significantly reduced barrier to uploading data to PCTS (automatically available from our dashboard) if the data is synced on time by the ANM. They can focus in on filling the missing gaps with secondary data sources for mothers who are incomplete and turn an eye to data quality – both saving their time and empowering them to make a bigger impact if a health center needs follow-up on patient data.
- We are enabling automated data quality checks. We can see how long it takes for each patient to be created, where the patient was created, by which ANM, and also screen for anomolous data and data trends that suggest manipulation. Our monitoring team can directly address the nurses who may be manipulating the data, or falsifying data outside the camp setting. We also have plans to integrate biometric authentication along with houeshold spot checks, so that the ANM may not keep the pendants to herself.
- We are giving the health administrator the training alongside the tools to make use of the Dashboard every week to engage with their workforce directly. It’s our goal to give them the visibility to see which vaccine vial made it to which child at the last mile at what time and by which nurse – alongside the auditable digital proof to confirm the data they see in the form of the Necklace.
Of course some barriers will remain, and we have to work around them. One of the biggest barriers lies in the government mandate to keep paper proofs on the RCH register and MAMTA card for each health interaction. It has been shown time and time again, that providing the ANM with another set of data entry tasks causes the intervention to suffer greatly.
Our strategy is multi-pronged:
- We are seeking approval from the State MoH in Rajasthan to directly upsync our data from the KB backend to the PCTS server, and get permission/mandate to remove the burden of paper proofs in our pilot camps
- We are providing printable formats that could serve as the paper proofs required on the RCH register and MAMTA card
- We are backfilling data with the ANMs
- We are providing the ANM with a “Search Patient” and “Camp Summary” feature. At the end of the camp she can use the Camp Summary feature to inform the ASHA about follow-up steps for specific mothers who both attended and failed to attend for the given week. With Search Patient, the ANM can independently and comprehensively fill the RCH register on the off-camp day, by selecting the week, the health camp, and the patient – to see the entire log of health related details. Our goal is for the ANM to only use the KB system and perhaps fill out the MAMTA card at the point of care. As India shifts towards digitization and away from paper records and the leakages theirin, we expect the MAMTA card to soon become obsolete with the KB necklace coming in its stead
The thread that ties it all together is an effort to support and engage all members of the health value chain with a sense of ownership. The value of this support is critical to the success of our project.
It is fair to ask, whether the cost of such our technological solution is significantly overshadowed by the underlying cost of implementation – and we say this despite the tremendous amount of investment we make and obstacles we overcome to provide the most robust technology to the ANMs. From our experience with Seva Mandir, we learned that the field monitors who were assisting the nurses and filling the records on the KB 1.0 application provided a vital pillar of support at the camp site. This type of extra assistance needs to be factored into the cost of conducting each camp. While AHSAs do provide support, and often are very knowledgeable about the patient cohort, it is very possible that the ANM needs a scribe or data entry person at the point of care itself. This could free the ANM to actually conduct the required pregnancy tests for example while simultaneously improving data quality for antenatal through infant care outcomes. If a Data Entry Operator were to be paid 600 INR (around $9/ camp day) to enter the data of 20 camp attending patients with the KB system – the additional cost per patient would come to $0.45 per patient visit (mother or child). Assuming 4 visits per mother/child on average, the incremental cost would come out to under $2 per patient. Some of this cost could be captured by simply rearranging the work schedule of DEOs who are otherwise taxed with the long process of translating nures handwriting into the PCTS access database. Finally, the benefit could be a radically transformed rural health information system, where a single point of data collection lives at the point of care.
Moving forward, we are encouraged by India’s pivotal steps towards digitization and transparency. We hope that this momentum can form a nexus with India’s recent declaration to join the UN Network to improve the quality of maternal and child health care. The WHO clarified the significance of this paradigm shift:
“Attention is now shifting from access to care to improving the quality of care so that countries can achieve the Sustainable Development Goals targets and to end preventable maternal, newborn, and child deaths by 2030.”
We strongly believe that improving the quality of data collected at the point of care is a necessary step for improving the quality of care. It means we can better engage with patients at high risk and seek out patients who are falling through the cracks.
We will continue to learn with open minds by rooting ourselves in the communities we hope to serve.
We will continue to push forward by focusing on the process.