Mapping the Practical Turn: A Digital Pre-History of the Philosophy of Mathematical Practice
Max Noichl, Gareth R. Pearce & Charles H. Pence
2024-08-22
Outline
- Idea behid the project
Problems!
How to represent texts? – Topic modelling runs into all kinds of problems!
A lot of older texts are messy in pretty unforseable ways. How to do cleanup?
How to identify philosophy of math? Little cohesion, similar, but different groups (philosophy of physics, philosophical logic)
Problems with BOW & topic-modelling
- Hard to deal with poly-semi
- These problems get worse, if you expect only subtle topical differences.
- Heavy information-loss.
- Answer: Embedding-LLMs! (BERT, RoBerta, Deberta, etc.)
- Problem: Out of the box models don’t know philosohy.
- Answer: Finetune!
The Data
- Large sample size: JStor + WOS (283,744 articles after filtering).
- Refined by journal tags & PhilPapers journal list.
- Time range: 1900-2021.
- Interactive data-cleaning strategy.
- Sample extended to include PhilPapers abstracts.
Network epistemology
- Traditional epistemology focuses on individual rationality
- What is the proper response to evidence?
- Network epistemology can target the structure of communication
- Which communication structures are best?
- Kicked off by Zollman (2007) who uses agent-based modelling and simulations
Recap: Zollman (2007)
- Agents (scientist) evaluate two competing methods, with similar, but slightly different quality. (Bandit-problem mirroring clinical trials). They communicate their evidence on a network.
- Agents cease evaluating methods they believe to be inferior.
- Main findings:
- Less connectivity can lead to more reliable groups – Community structure matters!
- There is a tradeoff between speed of convergence and reliability
“Even beyond the problem of maintaining the division of cognitive labor, this model suggests that in some circumstances there is an unintended benefit from scientists being uninformed about experimental results in their field. This is not universally beneficial, however.
In circumstances where speed is very important or where we think that our initial estimates are likely very close to the truth, connected groups of scientist will be more reliable. On the other hand, when we want accuracy above all else, we should prefer communities made up of more isolated individuals.” – Zollman (2007)
Following these strands in Network Epistemology
- A strand on robustness:
- Robustness under changes in parameter settings (Rosenstock, Bruner, and O’Connor (2017)),
- Robustness under changes in modelling choices (Kummerfeld and Zollman (2016); Frey and Šešelja (2018); (freyRobustnessIdealizationsAgentBased2020?))
- A strand on conformity: (Zollman (2010b); Mohseni and Williams (2021); Weatherall and O’Connor (2021); Fazelpour and Steel (2022))
- A strand on epistemically impure agents (including financial interests) (Holman and Bruner (2015); Weatherall, O’Connor, and Bruner (2020))
- Our work: Empirically guided robustness tests.
Main network-types used in Zollman (2007)
Convergence as a function of network-size – Rosenstock, Bruner, and O’Connor (2017)
“As a result, we cannot say with confidence that we expect real world epistemic communities to generally fall under the area of parameter space where the Zollman effect occurs. We are unsure whether they correspond to this area of parameter space, or some other area, or some other models with different assumptions.” – Rosenstock, Bruner, and O’Connor (2017)
But what are the appropriate
networks to test for the effect?
Approach 1: Artificial Networks
Some candidates for more realistic networks
But how to evaluate the influence of specific network structures?
Rewiring!
Changing network-structure through randomization
(E. g. Watts-Strogatz-graph)
Parameters
| Parameter | Type/Range | Description |
|---|---|---|
| Number of Agents | 11 to 200 | Number of agents in the network |
| BA-Degree | 2 to 10 | Degree for the Barabási-Albert (BA) model |
| ER-Probability | 0 to 0.25 | Probability for edge creation in the Erdos-Renyi (ER) graph model |
| Rewiring probability | 0 to 1 | Probability of rewiring in the network generated |
| Uncertainty | 0.001 to 0.01 | Probability-difference between the theories. (Smaller: harder problem) |
| n-experiments | 10 to 100 | Number of experiments to run each round (Smaller: Less information collected) |
| Network-type | ‘ba’, ‘sf’, ‘ws’ | Type of network (‘ba’: Barabási-Albert, ‘sf’: directed Scale-Free, ‘ws’: Watts-Strogatz) |
| Agent-type | ‘bayesian’, ‘beta’ | We currently implement two agent types: The original bayesian one, and a beta-distribution based Thompson-sampler. |
Results: Bayesian-learner & Barabási-Albert: In nearly all simulations, basically all agents learn the correct method.
Model-fit: Bayesian-learner & Barabási-Albert
=============================================== ==========================================================
Distribution: NormalDist Effective DoF: 19.5321
Link Function: IdentityLink Log Likelihood: -298574.6004
Number of Samples: 990 AIC: 597190.265
AICc: 597191.178
GCV: 0.0017
Scale: 0.0016
Pseudo R-Squared: 0.145
==========================================================================================================
Feature Function Lambda Rank EDoF P > x Sig. Code
================================= ==================== ============ ============ ============ ============
Probability of Rewiring [0.6] 6 4.2 7.09e-01
Uncertainty Level [0.6] 6 3.2 2.88e-02 *
Number of Experiments [0.6] 6 3.2 6.98e-01
Mean Degree [0.6] 6 3.3 1.11e-16 ***
BA-Degree [0.6] 6 2.6 1.11e-16 ***
Number of Agents [0.6] 6 3.0 1.50e-05 ***
Intercept 1 0.0 1.11e-16 ***
==========================================================================================================
Significance codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Predicting Quality: Messing with the hierarchical-ness of the network (prob-rewiring) doesnt seem to make much difference, when predicting the share of correct agents at convergence.
Predicting Speed: Probability of rewiring also doesn’t influence convergence-time, which is determined by the usual suspects (problem difficulty, number of experiments, degree)
These results are very similar for all tested network-types!
We don’t find an reliability
advantage for sparser networks.
We don’t find an influence
of rewiring network-structure.
Approach 2: Real Networks
Motivation:
- There are several examples of sub-optimal processes in the history of science.
- “The hypothesis that peptic ulcers are caused by bacteria did not originate with Warren and Marshall, it predates their births by more than 60 years. But, unlike other famous cases of anticipation, this theory was the subject of significant scientific scrutiny during that time. To those who have faith in the scientific enterprise, it should come as a surprise that the widespread acceptance of a now well supported theory should take so long.” – Zollman (2010a)
- Our current Examples: Peptic Ulcer (n= 133403, – 1978) & Perceptron (n= 3519, – 1979)
- Author-based citation-network collected from OpenAlex
The Perceptron network
Degree-distribution of the perceptron-network (n=3519).
The Perceptron network (randomly rewired)
Degree-distribution after random rewiring (p=.2), moving towards a normal degree distribution. Rewiring does not change the mean degree.
Results: Perceptron - Quality
Share of correctly informed (bayesian) agents at convergence depending on varied parameters.
Results: Perceptron - Quality
Isolated dependencies of the correctness of agents on varied parameters. Probability of rewiring seems to strongly drive outcomes!
Results: Perceptron - Speed
Results: Perceptron - Speed
Tentative Results
Using more sophisticated
network-models doesn’t end
original robustness worries.
But network-structure clearly does matter,
as we find real, suboptimal networks!
Discussion
- Why does randomization affect our empirical, but not our artificial networks?
- Are there better network-models that reproduce this effect, e. g. more clustered ones?
- What about other rewiring techniques (e.g. increasing pref. attachement)?
- How to adequately represent empirical networks?
- Vary other aspects of networks – E. g. connectivity (c.f. Zollman (2007), 3.1) vs. degree-inequality?
