Bookmarks (704)

Published on May 2, 2025 3:55 PM GMTDespite being fairly well established as a discipline, genetics is a science that has yet to reach its potential, both policymakers and the general population are extremely skeptical of it and, as a consequence, our society has setup several barriers to prevent its flourishing.In this post I will try to imagine what the potential benefits of embracing...

Published on May 2, 2025 5:49 PM GMTMy paper "AI Welfare Risks" has been accepted for publication at Philosophical Studies!I argue that near-future AI systems may have welfare, that RL and behaviour restrictions could harm them, that this poses a partial tension with AI safety concerns, and I propose three tentative AI welfare policies AI labs could implement to reduce such welfare risks.Building on...

Published on May 2, 2025 3:27 PM GMTNarmeen developed, ideated and validated K-steering at Martian. Luke generated the baselines, figures and wrote this blog post. Amir proposed the research direction and supervised the project. The full interactive blog will be available closer to the publication of the complete paper on the Martian website.TL;DR: We introduce K-steering, a steering method for language models that allows...

Published on May 2, 2025 2:19 PM GMTThe video is about extrapolating the future of AI progress, following a timeline that starts from today’s chatbots to future AI that’s vastly smarter than all of humanity combined–with God-like capabilities. We argue that such AIs will pose a significant extinction risk to humanity.This video came out of a partnership between Rational Animations and ControlAI. The script...

Published on May 2, 2025 1:10 PM GMTRight before releasing o3, OpenAI updated its Preparedness Framework to 2.0. I previously wrote an analysis of the Preparedness Framework 1.0. I still stand by essentially everything I wrote in that analysis, which I reread to prepare before reading the 2.0 framework. If you want to dive deep, I recommend starting there, as this post will focus...

Published on May 2, 2025 12:27 PM GMTSeveral ex-employees of OpenAI filed an amicus brief in the Musk v OpenAI[1] case. This proposed brief argues that OpenAI should not be allowed to move from its existing nonprofit structure to a for-profit structure. We now have an order on the motions to dismiss in the case. This order also seems to deny the motion for leave...

Published on May 2, 2025 2:58 AM GMTNote: I didn't write this essay, nor do I own the blog where it came from. I'm just sharing it. The essay text is displayed below this line.The continuum fallacy is to deny the meaningfulness of discrete categories, just because they are a somewhat arbitrary partition of a continuum. More generally, it is to deny the meaningfulness...

Published on May 2, 2025 10:29 AM GMTOn a theoretical road, the number of cars traveling is proportional to the speed of each car, so that the total number of motorists is constant regardless of speeds.Assuming all cars are traveling at a speed that gives 3 seconds of time between cars, any change to speed limit cannot affect the traveler throughput, and each car...

Published on May 1, 2025 11:08 PM GMTThis is post 3 of a sequence on my framework for doing and thinking about research. Start here. Thanks to my co-author Gemini 2.5 ProIntroductionSpend enough time around researchers, and you'll hear talk of "research taste." It's often presented as a somewhat mystical quality distinguishing the seasoned research from the novice – an almost innate sense for...

Published on May 1, 2025 10:46 PM GMTWe’re excited to release a new AI governance research agenda from the MIRI Technical Governance Team. With this research agenda, we have two main aims: to describe the strategic landscape of AI development and to catalog important governance research questions. We base the agenda around four high-level scenarios for the geopolitical response to advanced AI development. Our...

Published on May 1, 2025 2:00 PM GMTGemini 2.5 Pro is sitting in the corner, sulking. It’s not a liar, a sycophant or a cheater. It does excellent deep research reports. So why does it have so few friends? The answer, of course, is partly because o3 is still more directly useful more often, but mostly because Google Fails Marketing Forever. Whereas o3 is...

Published on May 1, 2025 1:54 PM GMTIt'll take until ~2050 to repeat the level of scaling that pretraining compute is experiencing this decade, as increasing funding can't sustain the current pace beyond ~2029 if AI doesn't deliver a transformative commercial success by then. Natural text data will also run out around that time, and there are signs that current methods of reasoning training...

Published on May 1, 2025 1:50 PM GMTIt is often noted that anthropomorphizing AI can be dangerous. People likely have prosocial instincts that AI systems lack (see below). Assuming AGI will be aligned because humans with similar behavior are usually mostly harmless is probably wrong and quite dangerous.I want to discuss a flip side of using humans as an intuition pump for thinking about...

Published on May 1, 2025 11:10 AM GMTMotivation: Improving group epistemics.TL; DR (Changes to) P doom/alignment difficulty are a shibboleth dominating conversations, distorting epistemics. Instead, focus on updates to your gears level models. Focus on near and concrete details instead of far and vague abstractions. People frequently opine on whether some alignment news is good or bad for alignment. "Training AI on insecure code makes...

Published on May 1, 2025 2:00 AM GMT I recently read a blog post that concluded with: When I'm on my deathbed, I won't look back at my life and wish I had worked harder. I'll look back and wish I spent more time with the people I loved. Setting aside that some people don't have the economic breathing room to make this kind...

Published on May 1, 2025 12:33 AM GMTTo make frontier AI safe enough, we need to "lift up the floor" with minimum safety practicesAnthropic has popularized the idea of a “race to the top” in AI safety: Show you can be a leading AI developer while still prioritizing safety. Make safety a competitive differentiator, which pressures other developers to be safe too. Spurring a...

Published on April 30, 2025 11:43 PM GMTFormal Systems Create CoordinationWe live in a mesh of customs, norms, and formal laws nudging our behaviour. Some of these systems are simple and intuitive, such as the customary respect for others’ property and refrain from violence. Others like the patchwork of taxes buried in the price of everything we buy are formalised, complex, and require experts...

Published on April 30, 2025 11:03 PM GMTAs an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called...

Published on April 30, 2025 8:20 PM GMTWhoops. Sorry everyone. Rolling back to a previous version.Here’s where we are at this point, now that GPT-4o is no longer an absurd sycophant. For now.Table of ContentsGPT-4o Is Was An Absurd Sycophant.You May Ask Yourself, How Did I Get Here?.Why Can’t We All Be Nice.Extra Extra Read All About It Four People Fooled.Prompt Attention.What (They Say)...

Published on April 30, 2025 7:58 PM GMTThis time around, I'm sharing a post on Interconnects on why he doesn't believe that the AI 2027 scenario by @Daniel Kokotajlo and many others will come true, and he has 4 sections on this plus a bonus section below:1. How labs make progress on evaluations2. Current AI is broad, not narrow intelligence3. Data research is the...

Published on April 29, 2025 5:07 PM GMTJohn: So there’s this thing about interp, where most of it seems to not be handling one of the standard fundamental difficulties of representation, and we want to articulate that in a way which will make sense to interp researchers (as opposed to philosophers). I guess to start… Steve, wanna give a standard canonical example of the...

Published on April 29, 2025 4:13 PM GMTWelcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.In this edition: Experts and ex-employees urge the Attorneys General of California and Delaware to block OpenAI’s for-profit restructure; CAIS announces the winners of its safety benchmarking competition.Listen to the AI Safety Newsletter for...

Published on April 29, 2025 3:45 PM GMTI made another biology-ML podcast! Two hours long, deeply technical, links below.I posted about others ones I did here (machine learning in molecular dynamics) and here (machine learning in vaccine design). This one is over machine learning in protein design, interviewing perhaps one of the most well-known people in the field. This is my own field, so...

Published on April 29, 2025 3:26 PM GMTInference CertificatesAs a prerequisite for the virtuality.network, we need to enable organizations which host inference workloads to prove the following about a particular AI output:It was generated during this time period.It was generated in this geographical region.It was generated using this unique chip.The generation workload was appropriately monitored.In addition, to make it as easy as possible for...

Published on April 29, 2025 1:10 PM GMTPreviously: #1, #2, #3. As time goes by, the fundamental things in life are still the same, and yet they change quite a lot with the times. But they don’t yet change so fast that the previous three editions of this are invalid. AI isn’t transforming the world that quickly, not yet. In the meantime, there’s always...

Published on April 29, 2025 9:50 AM GMTOn Thursday 1 May 2025 1900 (UK time) I'm giving a talk to discuss this post at Fazenda UK 9 Harrow Place, E1 7DB -- 10 minutes from Liverpool St station (note there are 2 other places with the same name nearby).The TL;DR from that post:TL;DR: This document is a copy and addition to this LessWrong post...

Published on April 29, 2025 2:00 AM GMTThis is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and...

Published on April 28, 2025 8:52 PM GMTIntroductionAI systems are advancing rapidly.At the same time, concerns about safety, alignment, robustness, and governance are becoming increasingly urgent.Building expertise in these areas is critical — especially within growing AI ecosystems like India’s.To address this need, AI Safety India is launching the first phase of its education program:Fundamentals of Safe AI (Phase 1 – Global Cohort).Applications are...

Published on April 28, 2025 11:46 PM GMTIntroductionFocusmate changed my life. I started using it mid-2023 and have been a power user since 2023. Here are the high-level stats:I've spent 2,456 hours on Focusmate.I've met 1,447 people.I have 10+ friends with whom I've spent between 30 and 300 hours.I've met five of my Focusmate friends in person. I've had a Focusmate's family over for...

Published on April 28, 2025 8:56 PM GMTIn most democracies,the very way people file complaints tells you how well their government works.You can vote.You can sue.You can organize, lobby, or protest.But in China?You write a physical letter to Beijing,then pray someone reads it.This is called Xinfang — the so-called "petition system."It sounds like democracy.It’s not.1. There is No Real "Letter Culture" in ChinaIn the...

Published on April 27, 2025 5:08 PM GMTI am starting a meetup for anyone in Victoria. I am unsure if there is anyone here who's into LessWrong but if you are RSVP!Discuss

Published on April 27, 2025 4:17 PM GMTOur universe is probably a computer simulation created by a paperclip maximizer to map the spectrum of rival resource‑grabbers it may encounter while expanding through the cosmos. The purpose of this simulation is to see what kind of ASI (artificial superintelligence) we humans end up creating. The paperclip maximizer likely runs a vast ensemble of biology‑to‑ASI simulations,...

Published on April 27, 2025 4:11 PM GMTI wasn't on LessWrong 10 years ago, during the pre-ChatGPT era, so I'd like to hear from people who were here back in the day. I want to know how AI-related discussions have changed and how people's opinions have changed.Are you leaning more towards "alignment is easy" or towards "alignment is hard", compared to many years ago?...

Published on April 27, 2025 3:31 PM GMTSo this post is an argument that multi-decade timelines are reasonable, and the key cruxes that Ege Erdil has with most AI safety people who believe in short timelines are due to the following set of beliefs:Ege Erdil don't believe that trends exist that require AI to automate everything in only 2-3 years.Ege Erdil doesn't believe that...

Published on April 27, 2025 8:18 AM GMT ChatGPT recently introduced improved cross-chat memory, and I've found myself using it a lot more sense then. Everything from helping write components with unfamiliar JavaScript libraries, to counseling me through a breakup. So much so that when I face a new problem, my mind turns not to breaking it down myself, but how to structure it as...

Published on April 27, 2025 2:38 PM GMTThis is post 2 of a sequence on my framework for doing and thinking about research. Start here.Before I get into what exactly to do at each stage of the research process, it’s worth reflecting on the key mindsets that are crucial throughout the process, and how they should manifest at each stage.I think the most important...

Published on April 27, 2025 2:08 PM GMTA common idea held by people debating AI art is that the growing amount of AI-generated images present in training data will cause AI diffusion models, like midjourney or stable diffusion, to produce bad results as existing flaws get amplified.However, I believe the opposite will occur, as there is a bias in AI outputs being published on...

Published on April 27, 2025 12:15 PM GMTSomeone asked me for an example of Recursive Self Improvement (RSI). I tried writing a sketch of a possible future where things go bad because of AI RSI. Here is my contribution to the Sci-Fi genre: ASI cautionary tales... Let's suppose some AI company creates a system with the goal of autonomously improving it's ability to sway...

Published on April 27, 2025 4:51 AM GMT"Hive mind" may not be the ideal term. What I'm really envisioning is something modelled on the existing relationship between human users and frontier LLMs, but with the "users" actually just being LLMs too, or simple programs that mediate between LLMs. But first, consider OpenAI's o-series models, and their human user base. For any particular model, OpenAI presumably...

Published on April 27, 2025 4:31 AM GMTDario Amodei posted a new essay titled "The Urgency of Interpretability" a couple days ago.Some excerpts I think are worth highlighting:The nature of AI training makes it possible that AI systems will develop, on their own, an ability to deceive humans and an inclination to seek power in a way that ordinary deterministic software never will; this...

Published on April 25, 2025 4:02 PM GMTIntroductionEffective Altruism (EA) is a social movement that aims to use reason and evidence to help others as much as possible. It encourages people to ask not just “how to do good”, but how to do the most good. This has led members to support things like global health interventions, existential risk reduction, and animal welfare.I used...

Published on April 25, 2025 3:32 PM GMTSince the 1970s, productivity has outpaced wage growth in many sectors. One sector where wages have kept up with productivity growth is tech, especially among programmers. But, I suspect this is changing thanks to the use of AI coding assistants.To quickly review the relevant economic history, before 1973 wage growth roughly tracked productivity in Western economies, but...

Published on April 25, 2025 2:40 PM GMT At work we've recently been using zstd as a better-compressing alternative to gzip, and overall I've been pretty happy with it. A minor documentation gripe, though, is that the behavior around multithreaded compression is a bit unclear. I understand it's chunking the work and sending chunks to different threads to parallelize the compression process, and this...

Published on April 25, 2025 10:03 AM GMTLetters to attorney generals, etc, to block OpenAI from moving assets into a corporation.Sep: Public CitizenDec: MetaJan, Apr: Californian civil society nonprofitsApr: Coalition for AI Nonprofit IntegrityApr: Safety ex-employees' brief for Musk lawsuitApr: Safety+ researchers on 'not for private gain' Discuss

Published on April 25, 2025 5:12 AM GMTI was recently encouraged to read Jeffrey Heninger's report "Why Did Environmentalism Become Partisan?"  It was interesting, but I thought it had some critical flaws.  I would've recommended rejecting it if I were reviewing it for an academic conference.  I've written a mock review below.  As typical when reviewing for a conference, I didn't aim to mince...

Published on April 24, 2025 9:22 PM GMTTLDR: I really like the graph where they show the best model for every price point. And how Google's models have the best performance at any price point. So I made it live and fetches news data and model regularly because things are moving fast and images don't refresh themselves. (Thanks Claude) https://winston-bosan.github.io/llm-pareto-frontier/(Will be hosted soon in a...

Published on April 24, 2025 9:15 PM GMTIn this post, we study whether we can modify an LLM’s beliefs and investigate whether doing so could decrease risk from advanced AI systems.We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all but the most implausible beliefs. We also demonstrate...

Published on April 24, 2025 9:15 PM GMTYesterday, I couldn't wrap my head around some programming concepts in Python, so I turned to ChatGPT (gpt-4o) for help. This evolved into a very long conversation (the longest I've ever had with it by far), at the end of which I pasted around 600 lines of code from Github and asked it to explain them to...

Published on April 24, 2025 8:17 PM GMTWhat in retrospect seem like serious moral crimes were often widely accepted while they were happening. This means that moral progress can require intellectual progress.[1] Intellectual progress often requires questioning received ideas, but questioning moral norms is sometimes taboo. For example, in ancient Greece it would have been taboo to say that women should have the same political...

Published on April 24, 2025 7:09 PM GMTIn late 2019 I wrote a TTRPG.The theme was alien sentience (or perhaps sapience is the technical term), both "organic" (extra terrestrial) and "artificial" (AI/robots). This is the prologue that kicks off play. I found it really fascinating to look back on late 2019 and how I was thinking about some of these topics. Earthdate March 2022. Two months ago...