[This article was first published on Data Analytics and AI Archives - Giles, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.TLDR: The last post identified potential transition pathways for truck drivers based on how closely other occupations matched them on skills, abilities and knowledge. This post follows the same basic approach across a wider range of occupations and asks whether two proxies could stand in for the ratings where no O*NET exists: a small language model comparing the meaning of job descriptions and the distance between two jobs’ US Standard Occupational Classification (SOC) codes. Both agree with the skill-based scores on average and appear to complement each other. The SOC code helps separate jobs in different classes of work and the descriptions are better at ranking occupations within similar groups. Together they explain about a third of the variation in the skill-based score overall, and 10 to 15 percent within a major group, which is where a real job seeker is likely to look. Neither is a substitute for the O*NET’s worker characteristic data, but both provide a reasonable sense check of transition pathway estimates which can be useful when it’s not possible to use data from the O*NET.BackgroundIn the last post I reproduced the similarity scores estimated by Bratanova et al. (2026). They used worker characteristic data from the O*NET to do this, with the basic idea being that two jobs that demand similar things of their workers should on average be easier to transition between. There’s of course more to their paper than this, but this is the crux of the approach.They’re certainly not the first to use the O*NET data in this way.1 There are also some great interactive tools2 that intuitively illustrate how the approach works. Unfortunately, only found these after I’d pored over the literature as an adviser on a similar project in India. Because the SOC didn’t map cleanly to India’s labour market3, we decided to dropped the O*NET in any case. But, the project left me with three questions that this post tries to answer:How much value does detailed worker characteristics data add when compared to similarity measures built from descriptive text, such as job descriptions?If the two approaches differ, are the differences significant enough to matter in practice?Given occupational classification systems are usually designed to group comparable jobs together, can they serve as proxy for skill gaps?Occupational Similarity: Three WaysHad we used the O*NET, answering these questions would have been relatively simple: calculate similarity scores using occupation characteristic from the O*NET and compare these with scores calculated from text-based measures and a numeric indicator of classification similarity, which is more or less what I’m going to do here.A primer on skill similarity and the O*NETIf you’re unfamiliar with the territory I’d recommend taking a look at my previous post where I explain why skill similarity measures are used for identifying transition pathways and the basic structure of the O*NET data that is drawn on. But, for those unwilling to leave this page:The O*NET: rates 800+ occupations across a standard set of characteristics that describe common abilities, skills and knowledge required to perform an occupation. Each characteristic is then given a rating based on the level required and its importance. Importance is meant to measure how critical a characteristic is to a job, whereas the level signifies the level of proficiency required / complexity of the task. For instance, the skill of speaking is important for both lawyers and paralegals, but lawyers are expected to have a higher level of speaking skills compared to paralegals (see here).Skill similarity as a proxy for viable transition pathways: When two occupations demand a similar set of abilities, skills and knowledge it’s assumed a worker would find it easier to transition between the two, on average. For instance, because the skills, abilities and knowledge required of truck drivers resembles bus drivers, it might be easier (and require less training) for them to transition between these jobs compared to becoming a data scientist. Both transitions are possible, but one involves overcoming more friction than the other, making the pathway less viable.4 The O*NET and US Standard Occupational Classification (SOC): The SOC uses a tiered classification system to group occupations, with jobs organized into 23 major groups, 98 minor groups and 459 broad occupations. Each SOC code specifies where a job has been assigned, for instance the SOC code for Chief Executives SOC is 11-1011.00, which places it in:Three ways to test whether two jobs are alikeBecause I’m interested in testing the ideas rather than describing them, I’m not going to cover each approach in too much detail, but the TLDR summary of each approach is below:Ratings similarity: occupations are compared based on the level and importance of skills, abilities and knowledge required to complete the job.Semantic similarity: Occupations are compared to one-another based on the similarity of job descriptions. Measurement is based on semantic similarity of job descriptions, which attempts to judge text based on its meaning rather than contents.SOC code distance: Differences in SOC code assignment is used to provide a naive measure of how similar two occupations are likely to be. Because occupations in the SOC are classified based on work performed and, in some cases, on the skills, education and/or training needed to perform the work5, it’s expected that similarly classified jobs will on average share similar characteristics (although this won’t necessarily be linear).Project set-upData:Like the previous O*NET post, analysis uses version 31.0 of the O*NET database. You can download the data used in this post here.Everything a reader might want to change sits in this one chunk: packages, paths, the rating scale bounds, the embedding model, the labels and the chart theme.library(tidyverse)library(janitor)library(readxl)library(text)library(scales)#where the data lives and where the figures get writtenref_dir_data ggplot(aes(x = major_b, y = major_a, fill = value)) + geom_tile(colour = "white", linewidth = 0.3) + scale_fill_gradient(low = ref_col_accent, high = ref_col_primary, name = "Average cosine similarity") + scale_y_discrete(limits = rev) + coord_equal() + ref_theme_heatmap + labs(x = "SOC major group", y = NULL)plt_sim_semantic_majorUse the SOC code distance as a “measure” of similarityI’ve used double quotes around measure to signify I’m making air quotes here, but to be clear: by measure I mean proxy. As even if similar jobs have been placed in the same group and groups are designed to correspond with wider groupings, such as white-collar, service, blue-collar and members of the military, code differences are not designed to precisely quantify where a job sits on some comparable continuum.To provide an example of what I mean by this: Surgical Assistants (29-9093) and Healthcare Practitioners and Technical Workers (29-9099) are next to each other on the SOC, with codes that are six apart from one another. Yet, Home Health Aides (31-1121) has a code that’s a little over 12 thousand apart from both jobs, despite the three jobs being broadly comparable.Although this is an extreme example, it does illustrate one of the core problems with relying on raw SOC codes as a proxy: small differences in assignments can result in large code differences that don’t correspond to how similar (or different) two jobs are. Added to this, even when these differences do provide a reasonable proxy for how similar (or different) two jobs are, this is unlikely to be constant across job pairs.Knowing all this, a reasonable person might ask why look at SOC codes at all. Four reasons:Data availability: Data as rich as the O*NET sometimes isn’t available, can’t be drawn on, or isn’t appropriate to use for the labour market being examined.A bad proxy can still have useful information: Occupation codes might get the ordering of similar jobs right, even when it gets the size of differences wrong.To complement other similarity measures: When data is scarce, classification code differences might provide useful information that other data sources lack, making it useful as part of a wider portfolio of measures, such as part of a composite index.As an intuitive sense-check: When adjacent codes tend to be used for similar jobs, score differences can provide a simple sanity check of other similarity scores.To limit the effect of adjacent occupations being assigned to different groups, the code below creates a difference measure based on an occupation’s ranking in the SOC, rather than raw code differences. This assumes that adjacent occupations are equally alike across the SOC, regardless of which groups they fall in. This approach was chosen as it feels easier to justify than relying on the raw codes, which sometimes exhibit large differences across classifications that are unlikely to correspond to how similar two jobs are.Given differences in SOC code rankings are linear, it’s not surprising that the heatmap presents limited variation. However, it does present two patterns that you might expect real similarity scores should follow* when mapped on a classification system like the SOC:Occupations pairs from the same group should exhibit higher similarity with one another; andSimilarity scores should decrease as the distance between groups increase.*(Provided the classification system groups jobs in a way that aligns with the nature of the work.)Note: The code also estimates raw SOC distance to allow testing whether within-group code differences have predictive power within the same minor group.#position of each occupation in the SOC list: 1 = first code, 894 = lastlkp_soc_rank arrange(soc_code) |> mutate(soc_rank = row_number()) |> select(soc_code, soc_rank)#two readings of the SOC for every pair:#1.) differences in code rankings#2.) differences in raw codessum_soc_distance distinct(soc_code_a, soc_code_b) |> left_join(lkp_soc_rank, by = join_by(soc_code_a == soc_code)) |> left_join(lkp_soc_rank, by = join_by(soc_code_b == soc_code), suffix = c("_a", "_b")) |> mutate( soc_rank_distance = abs(soc_rank_a - soc_rank_b), #the full code as one integer, suffix included: "11-9199.11" -> 11919911. #Removing the point as well as the hyphen keeps the suffix as whole digits, #so two specialisations of one code are 1 apart rather than 0.01, which #rounding and floating point would otherwise erase. soc_numeric_a = as.numeric(str_remove_all(soc_code_a, "[-.]")), soc_numeric_b = as.numeric(str_remove_all(soc_code_b, "[-.]")), soc_code_distance = abs(soc_numeric_a - soc_numeric_b), major_a = str_sub(soc_code_a, 1, 2), major_b = str_sub(soc_code_b, 1, 2) )#average list-position distance between every pair of major groups, self-pairs#excluded. The raw code is kept for later; drawn up here it would show the same#bands with different numbers.sum_soc_distance_major filter(soc_code_a != soc_code_b) |> summarise(value = mean(soc_rank_distance), .by = c(major_a, major_b))plt_soc_distance_major mutate(major_a = factor(major_a, levels = lkp_soc_major$soc_major, labels = lkp_soc_major$soc_major_label), major_b = factor(major_b, levels = lkp_soc_major$soc_major)) |> ggplot(aes(x = major_b, y = major_a, fill = value)) + geom_tile(colour = "white", linewidth = 0.3) + #the ramp is flipped here, so darker still means closer scale_fill_gradient(low = ref_col_primary, high = ref_col_accent, name = "Average difference in list position (darker = closer)") + scale_y_discrete(limits = rev) + coord_equal() + ref_theme_heatmap + labs(x = "SOC major group", y = NULL, title = "Average difference in SOC code ranking by major group")plt_soc_distance_majorDo the measures agree?But, do the three measures blend agree with one another?The code below takes all three measures and combines them into a single dataframe so individual similarity measures can be compared against one another. Groupings are specified to help communicate insights from exploratory analysis (and arguments with Claude) that aren’t shown here for the sake of brevity. In short, the pairs cover:Groupings based on where occupation codes differ: Such as job pairs that differ by major, minor, or detailed SOC groupings. This allows us to ask whether the explanatory power of compared measures change when job pairs are drawn from different classifications.Groupings where jobs are in the same SOC groups: Such as where compared jobs are in the same major, minor or SOC group. This is to see if the predictive power of measures change when job pairs are drawn from closer occupational groups.The second group comes from the exploratory analysis left out of the post, which pointed to the SOC distance measure’s explanatory power varying based on how differently two jobs were classified. Because the ranking ignores the size of classification differences within an occupational group, raw SOC code differences might be better at separating jobs in the same group than the ranking, which I wanted to test.#each pair is labelled by the first level at which its two codes differ. A pair#that differs at the minor group shares the major group above it; a pair that#differs only in the suffix is two specialisations of one detailed occupation.ref_labels_differ inner_join(rlt_sim_semantic, by = join_by(soc_code_a, soc_code_b)) |> inner_join(sum_soc_distance |> select(soc_code_a, soc_code_b, soc_rank_distance, soc_code_distance), by = join_by(soc_code_a, soc_code_b)) |> #each pair once, lower code first, self-pairs dropped filter(soc_code_a < soc_code_b) |> mutate( #distances flipped to closeness, so every measure reads higher = more alike soc_closeness = -soc_rank_distance, soc_code_closeness = -soc_code_distance, #"53-3032.00": major "53", minor "53-3", broad "53-303", detailed "53-3032" shared_depth = case_when( str_sub(soc_code_a, 1, 7) == str_sub(soc_code_b, 1, 7) ~ 4, str_sub(soc_code_a, 1, 6) == str_sub(soc_code_b, 1, 6) ~ 3, str_sub(soc_code_a, 1, 4) == str_sub(soc_code_b, 1, 4) ~ 2, str_sub(soc_code_a, 1, 2) == str_sub(soc_code_b, 1, 2) ~ 1, TRUE ~ 0 ), differ_at = factor(ref_labels_differ[as.character(shared_depth)], levels = ref_labels_differ) )Examining the number in each groupSubsetPairsBy the group the codes shareAll pairs399,171Same major group24,526Same minor group7,851Same broad occupation1,097Same detailed occupation233The code below outputs a frequency table for each group. An important characteristic of the groups shown in the table above is that most job pairs span different major groups.#one row per cut and level, holding the pairs that belong to itdta_pairs_subsets mutate( cut_by = fct_inorder(cut_by), subset = factor(subset, levels = c(ref_labels_differ, ref_labels_share)), pairs = map2(cut_by, depth, \(ref_cut, ref_depth) { if (ref_cut == "By where the codes first differ") { filter(rlt_pairs, shared_depth == ref_depth) } else { filter(rlt_pairs, shared_depth >= ref_depth) } }), nmb_pairs = map_int(pairs, nrow) )dta_pairs_subsets |> select(cut_by, subset, nmb_pairs)Job description vs. skill-based similarityThe plot above compares similarity scores estimated from job descriptions and the skill-based scores. The relationship is positive, indicating that on average the same job pairs tend to have higher similarity scores using either measure. But, the fit isn’t great, particularly for job pairs with low similarity scores (which account for a large share of job pairs). The glib conclusion is that both measures tend to agree when the similarity of two jobs can be picked up by both measures, but not so much otherwise.#20,000 random pairs, fixed by the seed so the figure is the same every render.#The same sample feeds the SOC figure below.set.seed(20260918)dta_sample_pairs slice_sample(n = 20000)plt_scatter_descriptions ggplot(aes(x = sim_semantic, y = sim_ratings)) + geom_point(colour = ref_col_desc, alpha = 0.1, size = 0.6) + geom_smooth(method = "loess", se = FALSE, colour = ref_col_text, linewidth = 1.1) + ref_theme_post + labs(x = "Semantic similarity of descriptions (cosine; more alike to the right)", y = "Ratings similarity (0-100)", title = "Ratings similarity against semantic similarity of descriptions, 20,000 random pairs")plt_scatter_descriptionsSOC distance vs. skill-based similarityThe code below uses a scatter plot to compare SOC code distance measures for job pairs with the similarity scores estimated from worker characteristics.Although I spent quite a bit of time digging into how the two measures correspond with one another (or don’t), I’ve dropped quite a lot of this analysis to keep the post short. Most of the analysis told the same basic story: both distance measures provide some* explanatory power across occupations, but most of their power comes from separating distinct classes of work from one another. The loess curve flattening at greater distances says the rest: once two jobs have SOC list differences of 300 or more, it has little to say about how alike they might be.*(I was actually surprised how much explanatory power the measure had.)#ggplot2 apparently can't give two facets different axis transformations, so the two#panels are drawn separately and placed side by side with patchworklibrary(patchwork)plt_soc_position ggplot(aes(x = soc_rank_distance, y = sim_ratings)) + geom_point(colour = ref_col_soc_rank, alpha = 0.1, size = 0.6) + geom_smooth(method = "loess", se = FALSE, colour = ref_col_text, linewidth = 1.1) + #reversed so closer pairs sit on the right scale_x_reverse(labels = label_comma()) + ref_theme_post + labs(x = "Difference in list position (closer pairs to the right)", y = "Ratings similarity (0-100)", subtitle = "SOC list position")plt_soc_raw ggplot(aes(x = soc_code_distance, y = sim_ratings)) + geom_point(colour = ref_col_soc_raw, alpha = 0.1, size = 0.6) + geom_smooth(method = "loess", se = FALSE, colour = ref_col_text, linewidth = 1.1) + scale_x_continuous(trans = compose_trans( "reverse"), labels = label_comma()) + ref_theme_post + labs(x = "Difference in raw code (closer pairs to the right)", y = NULL, subtitle = "SOC raw code")plt_scatter_soc as_tibble(rownames = "measure_a") |> pivot_longer(-measure_a, names_to = "measure_b", values_to = "correlation") |> mutate( measure_a = factor(ref_labels_measure[measure_a], levels = ref_labels_measure), measure_b = factor(ref_labels_measure[measure_b], levels = ref_labels_measure) ) |> #the matrix is symmetric, so show each pair once, below the diagonal filter(as.integer(measure_a) > as.integer(measure_b))plt_cor ggplot(aes(x = measure_b, y = measure_a, fill = correlation)) + geom_tile(colour = "white", linewidth = 1) + geom_text(aes(label = round_half_up(correlation, 2), #dark text on light tiles, light text on dark ones colour = correlation > 0.5), size = 4.5) + scale_fill_gradient(low = "white", high = ref_col_soc_rank, limits = c(0, 1), name = "Rank correlation") + scale_colour_manual(values = c(`TRUE` = "white", `FALSE` = ref_col_text), guide = "none") + scale_y_discrete(limits = rev) + coord_equal() + ref_theme_post + theme(panel.grid = element_blank(), axis.text.x = element_text(angle = 30, hjust = 1)) + labs(x = NULL, y = NULL, title = "Rank correlation between the four measures, all pairs")plt_corThree measures of the same hierarchy(?)The code below produces a table of average similarity scores by each measure based on job pair grouping differences.The table points to the SOC hierarchy doing a reasonable job of grouping similar jobs together. It also shows that most job pairs sit in different major groups and score a little above 50. It’s tempting to interpret this as indicating that the scores aren’t particularly good at differentiating jobs. But, this is actually what you’d expect of a database that’s designed to cover everything, rather than provide a representative snapshot of the labour market.Where the codes first differPairsRatings similaritySemantic similaritySOC rank distanceMajor group374,645550.20316Minor group16,675700.3429Broad occupation6,754740.3911Detailed occupation864770.483Suffix only233760.442#average of each measure by where the codes first differsum_by_level summarise( nmb_pairs = n(), sim_ratings = mean(sim_ratings), sim_semantic = mean(sim_semantic), soc_rank_distance = mean(soc_rank_distance), .by = differ_at ) |> arrange(differ_at) |> mutate(sim_ratings = round_half_up(sim_ratings, 0), sim_semantic = round_half_up(sim_semantic, 2), soc_rank_distance = round_half_up(soc_rank_distance, 0))sum_by_levelBut the table also points to an uncomfortable truth about all three measures: it becomes increasingly difficult to use similarity scores to differentiate jobs once they share the same major group. Whether a job pair is in the same minor group, broad occupation, or have differences in the last two digits of the SOC code, the average similarity score shows limited movement. The plot below illustrates this by presenting the entire distribution of each measure by each group. Although the average score rises as the classification narrows, the boxes overlap so much that the classification tells you little about how alike two jobs are once they share a major group. The score still varies inside each grouping, which points to similarity scores being best used as a means to rank alternative pathways, rather than to put a precise number on how similar two jobs are.Note: the association between similarity measures and SOC groups is not new, it’s a design principle of the system. The SOC classifies occupations based upon work performed, skills, education, training, and credentials8 , which are many of the same characteristics the skill based scores were based on. In addition, because managers are intentionally grouped with workers they manage and/or supervise, skill similarity is likely to be deeply nested in the hierarchy. In short, a skills-based score agreeing with a system designed around skills is exactly what you’d expect. It’s also why I felt a SOC code distance measure might provide a plausible proxy for similarity scores in the first place.dta_plt_level_dist transmute(differ_at, `Skill-based ratings` = percent_rank(sim_ratings), `Job descriptions` = percent_rank(sim_semantic), `SOC list position` = percent_rank(soc_closeness)) |> pivot_longer(-differ_at, names_to = "measure", values_to = "percentile") |> mutate(measure = fct_inorder(measure))plt_level_dist ggplot(aes(x = differ_at, y = percentile)) + geom_boxplot(fill = ref_col_grid, colour = ref_col_text, outlier.alpha = 0.03, outlier.size = 0.3, width = 0.6) + #the level means, which are the numbers in the table stat_summary(fun = mean, geom = "point", colour = ref_col_desc, size = 2.5) + scale_x_discrete(labels = \(x) str_remove(x, "^Differ ")) + facet_wrap(vars(measure), nrow = 1) + ref_theme_post + theme(axis.text.x = element_text(angle = 25, hjust = 1)) + labs(x = "Where the two codes first differ", y = "Percentile of the measure", subtitle = "All three measures rise with the hierarchy on average (gold points), and all three overlap heavily below the major group") plt_level_distThe explanatory power of descriptions appears to be steady across the hierarchyTo prove that I’m not siding with SOC distance measures as a proxy for similarity paths, the plot below presents insights resulting from a long series of arguments with Claude.Presented in the plot are rank correlation of each measure with skill-based scores based on how job pairs are grouped within the SOC. Notice that across the largest set of pairs, SOC distance measures agree more closely with the skill-based rankings. But, in every other group job descriptions do a better job of predicting skill-based rankings. Notice also that the raw SOC code doesn’t beat the list measure either, which points to the information lost by using the ranking being minimal, even when jobs are in the same narrow classification.Aside from this supporting the idea that the distance measure is proxying characteristics embodied by the SOC, it also points to job descriptions providing a more reliable proxy for skill similarity the deeper in the hierarchy the jobs being compared sit. And since that is where most workers are likely to look when they think about changing jobs, descriptions are likely to be the more useful measure for sense checking potential transition pathways.# Rank correlation of the ratings score with each measure, on one set of# pairs. Written as a function because it runs once per subset.fnc_cor_with_ratings select(cut_by, subset, nmb_pairs, result) |> unnest(result)sum_cor_by_subset |> mutate(across(where(is.numeric), \(x) round_half_up(x, 2)))plt_cor_by_subset ggplot(aes(y = subset)) + geom_vline(xintercept = 0, colour = ref_col_grid) + #the gap between the two measures in each row is the story geom_segment(aes(x = `List position`, xend = Descriptions, yend = subset), colour = ref_col_grid, linewidth = 2) + geom_point(aes(x = `Raw code`, colour = "Raw code"), shape = 21, fill = "white", size = 3.2, stroke = 1.1) + geom_point(aes(x = `List position`, colour = "List position"), size = 4) + geom_point(aes(x = Descriptions, colour = "Descriptions"), size = 4) + geom_text(aes(x = Descriptions, label = round_half_up(Descriptions, 2)), vjust = -1.2, size = 3, colour = ref_col_desc) + geom_text(aes(x = `List position`, label = round_half_up(`List position`, 2)), vjust = 2.1, size = 3, colour = ref_col_soc_rank) + #the pair count, at the left edge of each row geom_text(aes(x = -Inf, label = paste0(format(nmb_pairs, big.mark = ",", trim = TRUE), " pairs")), hjust = -0.1, size = 2.8, colour = "grey50") + scale_colour_manual( values = c(Descriptions = ref_col_desc, `List position` = ref_col_soc_rank, `Raw code` = ref_col_soc_raw), name = NULL, guide = guide_legend(override.aes = list(shape = c(16, 16, 21), fill = "white")) ) + scale_y_discrete(limits = rev) + scale_x_continuous(expand = expansion(mult = c(0.25, 0.08))) + #one panel per cut_by, each with its own five rows facet_wrap(vars(cut_by), scales = "free_y") + ref_theme_post + theme(panel.grid.major.y = element_blank()) + labs(x = "Rank correlation with the ratings score", y = NULL, title = "How each measure tracks the ratings, within each subset of pairs")plt_cor_by_subsetDoes the agreement hold across occupation groups?The plot below presents the rank correlation of each measure with skill-based scores by occupational group, where job pairs are in the same major group.Once again, description-based scores outperform the SOC distance measure in most groups, and where they don’t the gap is small. The clear exception is Architecture and Engineering, which plausibly stems from the SOC hierarchy separating engineers from the technicians and drafters who support them, a technical grouping that the description-based similarity scores largely miss.#pairs where both occupations sit in the same major group. Each pair belongs to#exactly one group, so no need to count both directions.sum_cor_by_group filter(shared_depth >= 1) |> mutate(major = str_sub(soc_code_a, 1, 2)) |> summarise( nmb_pairs = n(), `Descriptions` = cor(sim_ratings, sim_semantic, method = "spearman"), `List position` = cor(sim_ratings, soc_closeness, method = "spearman"), .by = major ) |> pivot_longer(c(Descriptions, `List position`), names_to = "measure", values_to = "correlation") |> mutate(major = factor(major, levels = lkp_soc_major$soc_major, labels = lkp_soc_major$soc_major_label))plt_cor_by_group ggplot(aes(y = major)) + geom_vline(xintercept = 0, colour = ref_col_grid) + geom_point(aes(x = correlation, colour = measure), size = 2.8) + #pair count at the left edge of each row, so the thin groups announce themselves geom_text(data = \(d) distinct(d, major, nmb_pairs), aes(x = -Inf, label = paste0(format(nmb_pairs, big.mark = ",", trim = TRUE), " pairs")), hjust = -0.1, size = 2.6, colour = "grey50") + scale_colour_manual(values = c(Descriptions = ref_col_desc, `List position` = ref_col_soc_rank), name = NULL) + scale_y_discrete(limits = rev) + scale_x_continuous(expand = expansion(mult = c(0.3, 0.05))) + ref_theme_post + labs(x = "Rank correlation with the skill-based score", y = NULL, title = "How well each proxy measure tracks the skill-based score for pairs within a SOC major group")plt_cor_by_groupComparing the predictive power of similarity measuresThe code below attempts to answer the question that inspired this post in the first place: if you don’t have detailed job characteristic data like the O*NET, how far might you get using national classification structures and job descriptions?In an attempt to answer this, the explanatory power of each measure is compared across the same five job pair groupings and a group with all job pairs. For each group, a linear regression model is used to estimate the explanatory power of measures individually and when combined together. A model with a higher adjusted R squared score explains a larger share of differences in skill-based score. The gap between the best measure and the combined model can be thought of as the marginal predictive value of adding the other measure.The results support the same stories that were presented before:Neither measure has much explanatory power for skill-based scores: A model that combines both measures at once can explain about a third of variations in skill-based scores across all pairs, but only 10 to 15 percent once two jobs are from the same major group.Both proxies appear to work as complements to one another:Note: p-values and standard errors have not been reported as the observations used in the model aren’t independent e.g. as the same job is included multiple times across job pairs. This isn’t expected to impact the r-squared scores, but will influence anything that relies on the number of independent observations.#the models, defined once: names, formulas and colours in the same orderref_models mutate(result = map(pairs, fnc_r_squared)) |> select(subset, nmb_pairs, result) |> unnest(result) |> mutate(subset = fct_relevel(subset, "All pairs"), model = factor(model, levels = ref_models$model))sum_models |> mutate(r_squared = round_half_up(r_squared, 3)) |> select(-nmb_pairs) |> pivot_wider(names_from = model, values_from = r_squared)plt_models ggplot(aes(y = subset)) + geom_vline(xintercept = 0, colour = ref_col_grid) + #the gap between the best single measure and the combined model is what the #second measure adds geom_segment(data = \(d) d |> summarise(x = max(r_squared[model != "Descriptions + SOC list position"]), xend = r_squared[model == "Descriptions + SOC list position"], .by = subset), aes(x = x, xend = xend, yend = subset), colour = ref_col_grid, linewidth = 2) + geom_point(aes(x = r_squared, colour = model), size = 4) + geom_text(data = \(d) distinct(d, subset, nmb_pairs), aes(x = -Inf, label = paste0(format(nmb_pairs, big.mark = ",", trim = TRUE), " pairs")), hjust = -0.1, size = 2.8, colour = "grey50") + scale_colour_manual(values = ref_col_models, name = NULL) + scale_y_discrete(limits = rev) + scale_x_continuous(expand = expansion(mult = c(0.4, 0.05))) + ref_theme_post + theme(panel.grid.major.y = element_blank()) + labs(x = "Share of variation in the skill-based score explained (adjusted R²)", y = NULL, title = "How much of the skill-based score each proxy recovers, for all pairs and by where the two codes first differ")plt_modelsWhat each measure bringsThe upshot of the regression results is that neither one of the measures can stand in as a substitute for skill-based similarity scores, but both measures appear to hold value in the absence of occupational data as detailed as the O*NET. A classification distance measure might say little about transition pathways for jobs that resemble each other, but might provide a useful metric for splitting occupations into distinct groups between which transition is less likely. From there, similarity scores based on job descriptions might be useful for ranking potential transition paths between similarly grouped job pairs.The analysis also points to the measures being practically useful for sense-checking similarity scores, which was what I wanted to test in the first place. For instance, where a country’s occupational classification standards align with the ILO’s International Standard Classification of Occupations, the broad structure of occupation codes should behave in a similar way to the SOC. And classification systems that include text-based information of each job and occupational group provide a means for producing semantic similarity scores. Neither measure is likely to be as rich as the O*NET. But, when the O*NET can’t (or shouldn’t be) used for analysis, both provide accessible metrics for validating transition pathways estimated from non-traditional and unstructured data sources, such as online job postings, vocational curricula and survey data.9Summing upThe post has its origins in a project to identify potential occupational pathways for a country with limited data and a labour force that looked nothing like most of the OECD. Neither problem is unique, but the comparability of local occupations to their US counterparts is a critically important consideration when deciding whether to use the O*NET, particularly when the pathways are intended to inform public policy.10 One solution to this is to leverage local data sources to develop a local database of occupational characteristics as a substitute to the O*NET. But, even if sufficient data and money exist to make this possible, it can be hard to know whether the identified pathways make sense. This post was meant to test whether description-based similarity measures and classification distance scores might serve as a basic sense check, which the analysis indicates they can. The classification distance measure might point to whether a pathway crosses a major occupational boundary that doesn’t make sense, the descriptions should help check if the rankings of transition pathways within a group look sane. Neither is likely to tell you that a particular pathway is correct, but together they will hopefully point to job transitions that are implausible.One of the things I argued with Claude about while finalizing this post, was its use of the word “cheap” to describe the two proxy measures. I didn’t like the phrasing (and I still don’t), but Claude is right that both measures are cheap. One is produced by subtracting one classification code from the another. The other comes from a transformer model that runs on my laptop, takes