AI companies are aiming to dramatically transform our world by developing machines that could surpass human intelligence, provoking significant economic changes. Sam Altman, CEO of OpenAI, has described this endeavor as creating a form of "magic intelligence in the sky, " termed artificial general intelligence (AGI). Altman suggests that AGI could disrupt capitalism and may represent a significant threat to humanity's continued existence. A critical question arises: Did anyone actually consent to this kind of AI?It raises concerns about the right of a few tech leaders to reshape the world without public input. The lack of democratic engagement in the creation of technologies that could fundamentally alter society is troubling, and even industry leaders express discomfort with this approach. Jack Clark, co-founder of Anthropic, voiced confusion about the absence of government oversight in AI development and questioned how much permission tech developers should seek from society before implementing irreversible changes. The conversation often encounters three main objections from tech enthusiasts. The first, “Our use is our consent, ” argues that the popularity of AI tools like ChatGPT signifies public approval.

However, this argument fails to recognize that user engagement does not equate to informed consent, as many users are unaware of the potential negative impacts of these technologies, such as their environmental cost. People often feel pressured to adopt tech due to professional competition and societal expectations. The second objection, “The public is too ignorant to tell innovators how to innovate, ” refers to the historical practice of innovators proceeding without public input. While some advancements did occur this way, the threats posed by technologies like AGI necessitate democratic oversight to prevent potential disasters. There are precedents for seeking public input for high-stakes technologies, like nuclear and biohazard regulations, underscoring that stakeholder engagement is crucial. The third objection, “It’s impossible to curtail innovation anyway, ” claims that technological progress is inevitable. This is a myth, as societies have successfully refrained from or restricted the development of certain technologies, such as human cloning and certain space activities. Historical examples, like the Asilomar Conference's moratorium on genetic experiments, illustrate how societal choices can shape technological trajectories. Ultimately, the creation of AGI and other powerful technologies should involve collective decision-making, as their impacts resonate across humanity, echoing the old Roman proverb: "What touches all should be decided by all. "