Pramit Bhattacharya of Mint has a piece on the topic. The article is based on a paper by Prof Prerna Singh.
The broad idea is that Kerala elites could promote sub-nationalism within the state and provide public goods in a more egalitarian fashion. UP elites were more nationalist in their approach and did not bother too much about the local population. The public goods provision was selective and narrow:
Singh shows the level of group solidarity within states, or the spirit of sub-nationalism, has a significant influence on social welfare policies of state governments. Other things (such as the level of inequality and political competition) remaining the same, the greater the spirit of sub-nationalism, the higher will be the commitment to improve health and educational outcomes, writes Singh.
She argues that in states where the elites have strived to foster a strong spirit of sub-nationalism, they try to ensure that the resources of the state are used in an egalitarian manner, and the worst-off sections of the state receive state benefits. Consequently, social outcomes tend to be better in such states.
This narrative is used to explain the divide between UP and Kerala:
In the mid-19th century, both regions that correspond to the present-day states of Kerala and UP, Travancore and the North-Western Provinces, respectively, were poor and divided, writes Singh. Caste divisions were more pronounced, and fractionalization was higher in Travancore than in the North-Western Provinces. Both regions were characterized by similarly low levels of social development in the mid-to-late 19th century. Female literacy was near zero and mortality rates high in both provinces. Yet, the trajectories of the two provinces differed sharply over the course of the past century, leading Kerala to move miles ahead of UP in development outcomes.
In the mid-19th century, a small minority elite—non-Malayali Brahmins in Travancore and Muslims in the North-Western Provinces—ruled the roost in both regions. Owing to a combination of several factors, other social groups began gaining economically in both provinces. These upwardly mobile but politically weak groups—Nairs, Syrian Christians and Ezhavas in Travancore, and Hindu merchant castes in the North-Western Provinces—came to demand political power commensurate with their improved socioeconomic status.
But the strategies of the challengers were different in the two regions. In Travancore, the varied groups came together under the banner of Malayali sub-nationalism to clip the wings of the non-Malayali Brahmins. Common cultural symbols came to the aid of the challengers. In the North-Western Provinces, the Hindu elites undermined common cultural bonds such as the shared language of Hindustani, and the Hindi-Hindu trope gained prominence to emphasize distinctiveness vis-à-vis the Urdu-Muslim group.
This history influenced the course of state politics even after Independence, and widened the gap between Kerala and UP substantially. A strong sense of shared identity and a conception of a shared destiny ensured that elites paid attention to an egalitarian distribution of state resources. A veteran communist leader from Kerala, E.M.S. Namboodiripad, described his party as the national party of Kerala. Even the local wing of the Congress party was not untouched by this spirit of sub-nationalism, and functioned relatively more autonomously than in UP, contends Singh.
The provision for public goods and merit goods such as health and education remained high in Kerala, irrespective of the party in power. In contrast, politicians in UP, who had never worked to foster a spirit of nationalism, were more interested in national affairs in the years immediately following Independence, and in catering to narrow caste interests. The provision of public goods and spending on social welfare, therefore, suffered in the state, argues Singh.
How all these political and social ideas are being used to figure the disparities in two regions. Actually, I would think that the idea could even be extended more backwards. Kerala was always in the game. KM Panikkar in his brilliant book explains how the Portuguese under Vasco Da Gama could not undermine the State of Travancore in warfare indicating the State’s superiority. The region also had huge geographical advantage and was pretty active in the trading game. There was a huge community advantage as well. Trade etc was always active in the area till British killed everything. Once British left, they could regain the focus more quickly. Kerala as a state was also much smaller.
But similar sort of advantage was not just there for United Provinces then and Uttar Pradesh now. There was never really any close community connect given the huge community disparity within the region. The social revolution could not take off. However, there was a time when UP had quite a few business clusters especially in the western region. Kanpur was once known as the Manchester of the east. Right from Meerut all the way to Lucknow there were clusters in handicraft items and items reserved for small scale industry. These sectors could have played a role in transforming the region. But once small scale reservations were lifted they just could not stand to competition.
So pretty complex and no easy answers….