(Unedited) Podcast Transcript 443: Planning for Generative AI

This week we’re chatting with David Wasserman of Alta Planning and Mike Flaxman of Heavy.AI about generative artificial intelligence. We chat about what generative AI is and how it is trained, and some of the ways it could be used or misused in a planning and transportation context.

To listen to this episode, visit Streetsblog USA or our hosting archive.

Below is an unedited by humans AI (ha!) generated transcript

Jeff Wood (1m 21s):
Well, David Wasserman, Dr. Michael Flaxman, welcome to the Talking Headways podcast.

David Wasserman (1m 41s):
Happy to be here. Jeff, happy to be here.

Jeff Wood (1m 42s):
Yeah, thanks for being here, both of you. We really appreciate it. Before we get started, can you tell us a little bit about yourselves? David, I know you’ve been on the show before and it’s been a little while, so perhaps you can refresh people’s memories a little bit as well.

David Wasserman (1m 53s):
Yeah. My name is David Wasserman. I’m Civic Analytics practice leader at Alta Planning and Design. I’m also chair of the APA Technology Division and an author of the planning advisory service memo on Artificial Intelligence in Planning practice, along with my coss speaker here, Mike? Yeah.

Mike Flaxman (2m 13s):
I am co-author with Esteem David of aforementioned Memo. My background is in biology originally and then in planning and I’m in my third career, so I’m now the Vice President of product at a startup in San Francisco that focuses on spatial analytics.

Jeff Wood (2m 29s):
Awesome. And Michael, how did you get interested in cities? You mentioned it’s the secondary thing career wise, but I’m wondering how you got into it.

Mike Flaxman (2m 36s):
I guess it goes back further. So I grew up in Reston, Virginia, which is one of the few planned communities in the United States and wasn’t really till I left it that I fully appreciated it ’cause it just felt normal to, you know, walk underneath the highway to get to my school. My father was interested in urban planning issues enough to kind of move from New York City out to the boondocks of Northern Virginia when I was a kid. Yeah. So I grew up with somebody that talked a lot about planning issues and eventually it caught up with me and I, I ended up getting a master’s degree in community and regional planning.

Jeff Wood (3m 9s):
And David, you too. Maybe we can rehash a little bit how you got into cities and planning.

David Wasserman (3m 13s):
Oh boy. I think for me it kind of started in high school where my initial motivation was actually mostly an interest in sustainable development and as I entered my college years and bachelors, it was recognizing that the how we operate and how we build and design our cities has massive influence on the outcomes from everything from public health to emissions to the impacts that we might have on the climate and as we enter an urban future. It was largely this understanding of the nexus between those and I think that’s how I really kind of came to the conclusion that this was an interesting area to be, or at least focus on.

Jeff Wood (3m 56s):
Last time we were here it was with Mattias talking about computer generated architecture, right, that’s right. For things like movies and city planning and episode 2 23. And I feel like today’s topic of Generative Aid AI specifically is an an extension of that. I kind of wanna know how you got into like that side of things, specifically thinking about the G I S and the three D and the same thing with you Michael as well.

David Wasserman (4m 18s):
Maybe I kind of feel like that was my second career to some extent, or my first, but the general trajectory there was there once I started working with City Engine as part of an Esri team and focusing on streets and how we build procedural models and enable people to build out three D complete streets quickly, you know, I was driven mostly by the problem and the technology was just one way to enable people to solve those problems in different places. And, and since then, even when there’s new tools out there, whether it’s Remix Streets or beyond typical City Engine still has a place and talking about transportation and land use together in in one area.

David Wasserman (4m 60s):
And yeah, I think I had a high school interest in three D art that just followed me throughout my career. You know, I haven’t stopped, I did the next best thing, I bought a three D printer and now I’m, my wife likes to say I’m printing garbage, but you know, I, I’m enjoying finding digital elevation models and printing them out. And same thing with three D streets. I’ve always been attracted to how we communicate in the third dimension.

Mike Flaxman (5m 23s):
I guess I would, I would say my story is similar in that I’ve also long been interested in that, but for myself, I started out from a biological motivation that I was a kind of a, I’m a lapsed biologist, but I, I didn’t wanna spend my career cataloging the decline of biodiversity. I wanted to actually work on, you know, things about urban forum and regional planning that would positively influence that and I lacked any actual artistic and design skills myself, but I had a knack for teaching computers how to do visualizations. And so, so I found myself at, you know, Harvard Graduate School of Design as somebody that couldn’t design their way out of a box with a pen, but I ended up making large scale visualizations.

Mike Flaxman (6m 7s):
So I visualized the San Pedro River Basin and its sustainable development and alternative futures. And so I came from that interest. I guess I always had a deep respect for design and design process, even though I lacked the skills to do it by hand myself. So the computer was my crutch and I took it as far as I could.

Jeff Wood (6m 27s):
I feel like I had that same crutch when I used to do maps myself. Well, let’s talk about Generative ai, shall we, I wanna know what Generative AI is. It’s kind of a tongue twister too, Generative AI and how it works. I’m wondering if you all can answer that question for me, because I’m still not quite sure.

David Wasserman (6m 42s):
When we talk about Generative ai, it’s this focus on algorithms that are specialized and models that are specialized in generating text images and even code, which is just another form of text to some extent.

Mike Flaxman (6m 56s):
Yeah, I guess I would, I would say it’s obviously a very new technology, but the word Generative is useful because it is generating novel things. So it’s quite different than traditional computing, which was kind of input output I think David mentioned, I I think one very important characteristic of it is that it’s inherently multimedia, right? And so this is kind of under the category of everything I learned in grad school is wrong. But when we used to do mapping and we’d always, you know, have to do a lot of work to get everything orthogonal for instance, because you couldn’t get any data from an image that wasn’t first ortho corrected and, you know, you couldn’t get any information out of speech and you couldn’t get any information out of video and or if you did, it was really, really difficult.

Mike Flaxman (7m 41s):
And so on the data sources side, the practical effect of this technology is that we can now compute on a whole variety of media types that we as humans use very expressively and have for centuries. But up until five years ago, you know, if it wasn’t in a tabular form like a spreadsheet or a database, your, your computational ability was really, really limited, right? And so all of that, or you had a tremendous amount of work to do to get something into a tabular format. And so to me, the pleasurable part about this moment is, is exactly that. But there’s this wider range of human expression that the computing world has been blind to that now we can start working with directly in these Generative models or using these Generative models.

Jeff Wood (8m 24s):
And so the models basically are taking the data, as it were, from images, from text, from research papers, et cetera. And how are the models created? I mean, what are they trained to seek out?

David Wasserman (8m 36s):
They’re very complex models and the type of model matters a lot when we’re talking about these types of discussions. But specifically for like a lot of the people talking about large language models and even some forms of these multimodal text models, it’s a focus on next word prediction where you have this input of a paragraph for example. And then parts of the paragraph might be one way to think about it is might be obscured and the model’s trying to infer what the, you know, filling in the gap might be. And it does that type of extrapolation further and further until we’re seeing the types of outputs that we’re seeing right now. Mike, would you describe it a different way?

David Wasserman (9m 18s):
’cause I’d be fine taking your description too.

Mike Flaxman (9m 20s):
Yeah, no, that’s, that’s a good start. And so on the text side that is indeed how it works. And then on the image side it’s very similar, right? So they, they mask portions of an image and then ask the computer to guess what should be underneath the black spot. And then they have reward function for guessing that correctly. And when you end up doing that, you end up with behavior that’s actually very similar to the human visual system. So you know, most of us walk around and we think that we have this complete three D model in our heads of the world, but if you look at people’s eye scans, you know, we’re constantly our, our eyes are darting around and gathering detailed information and we are filling in large parts of what we think we see.

Mike Flaxman (9m 60s):
And so, you know, in both cases with image, this approach to guessing missing parts of images and the same on missing parts of text, we’ve gotten to a point where mach, where machines can do a spookily good job at what humans have been doing for quite a while. And so that’s a little bit the mechanism and then a little bit of the consequence, we’ll, we’ll talk more obviously about the consequences, but in terms of mechanism, I guess I wanted to express it that way because the machines are, these are biologically inspired models, they’re not really biological but they’re biologically inspired and some of the behavior that we’re seeing out of these models then mimics human behavior in a fairly deep way.

Mike Flaxman (10m 42s):
So I don’t think that it’s just a parlor trick. I think that a lot of what these models are doing is imitating things that we as humans have done. Yeah. And the other part that I find very interesting to call out is that one of the consequences of the image models is that we’ve actually been able to separate style from substance within an image. And so these models can take, you know, a A three D visualization of the house and portray it in the style of multiple artists. Yep. And so to do that, which is completely unintuitive that you could do that, but that is an important characteristic that both aesthetics and kind of structural detail are kind of decomposed by these models.

Mike Flaxman (11m 24s):
And so there again, very similar to human beings, we can look at, you know, some crazy image by Van Gogh of the house where the, the walls are purple and the roof is yellow and yeah, we recognize it as a house and so does a machine learning model.

David Wasserman (11m 38s):
Yeah, similar things can happen in text for example. A lot of what is guiding these models as prompts, which are these text inputs that describe a desired output, if you say provide me a memo versus provide me a letter, which is more formal, it will identify characteristics that might be style dependent. But you could also give it text samples in a prompt and say, this is how I’ve written something previously. Please show me how you might write something pertaining to this subject in a different style. One of the more humorous examples for me was the original SWOT graphic that we put in in our presentation.

David Wasserman (12m 21s):
I asked a version of one of these models to create all those bullet points except written by a belter from the Expanse. And it, and it did a pretty funny approximation, well

Jeff Wood (12m 35s):
That’s what people have been doing. They’ve been putting in, you know, show us the family guy characters as Simpsons characters or yeah, those types of things. People have been very interestingly putting together their own prompts and figuring out how to artistically display different things in different ways. And I think that’s really interesting. The other interesting thing is the prompt kind of idea generally, I think, you know, previously to get something to work and we’re all kind of familiar with g i s, if you get something to work, you’d have to do it in a code language or you’d have to understand how the shape file works to combine it with a database and those types of things. But with the prompt for this specifically, and I think this makes it kind of more interesting for laypeople, is that you can just ask it a question and it doesn’t require all of these technical background details that you would’ve otherwise done before in something like G I ss.

Mike Flaxman (13m 22s):
Yeah, that’s absolutely true and I’m having a very strange moment at our startup where we’re working on English to sequel and the the strange moment is that the computer is now too creative about how it generates code. And so for all these decades, you know, where, where we as humans had to fit everything into slots to be, you know, highly literal and an extra period in your code would cause things to crash. We’re now at the point where the, the main interface to these models is actually a little bit too creative for some of the computer languages that we ourselves earlier invented. So we’re working our way through that actually training models internally to do a better job than the public models do.

Mike Flaxman (14m 6s):
Exactly on that point, to be able to follow G I ss commands not too creatively so that the underlying software can actually execute on the command. But I I find that a very strange moment in computer science basically, that we are too literal and the computer is too expansive, used to be on the other foot, but I think it’s also an interesting, you know, current limitation of the technology. I think that’s a hurdle that I think we’ll get beyond, but currently the models are really good at human language. They’re not quite as good at computer languages. They can be used for both, but they’re not quite as good at computer language yet.

Jeff Wood (14m 42s):
Well, yeah, we’ve seen recently somebody used an AI language model to, I think it was translate UNIFOR tablets in like three days versus like, it would, would take a, a human a thousand years or something to do all these tablets, which is an, an insane amount of processing that’s available out there. And I’m wondering, and I I wanna get to the transportation and planning portion of this in a second, but I’m wondering, you know, where we’re at with this technology, it’s kind of common to the common consciousness recently, but it’s had to have been worked on for quite a while and we’re in this moment now where it’s in the pop culture and it seems like people are saying that it will be able to do anything once it’s at its peak.

David Wasserman (15m 22s):
Yeah, I I mean back to the comment on like to the extent that it can handle what I would call structured problems, like, like we talked about g i s some of these technologies, there have been early forms of what we’ve called from a model like a called a GaN Generative adversarial network that has been able to help do things related to image creation, but they aren’t nearly as sophisticated as some of the models that are happening now. And also their scale is different as well. Some of these models are massive, like you need to have a one terabyte drive just to have it sit on the drive. And so there’s been this trend of the scale of the models increasing and so that’s a pseudo recent trend like say last couple years or so.

David Wasserman (16m 10s):
But also though, in terms of the realizing of the applications of these models, it’s very early days, you know, we’re starting to see papers come out like we talked about G i s where people are trying to create what they call solution graphs where they connect an L l M to some type of interpreter and have it do g i s commands based off of a high level task. But it’s only right 80% of the time. And so you can’t really do something in practice if you’re only right 80% of the time. There’s kind of this slot machine productivity effect when these models are used in practice where, you know, you put in your prompt crank the lever, hope something useful comes out, but it might be useful 60 to 80% of the time depending on what the problem is.

Mike Flaxman (16m 59s):
Yeah. The, the interface paradigm of having a conversation with your computer is fundamentally interesting because it, it, to me it hearkens back to the Socratic method, right? And when you say something that’s not specific, you may get a variety of results but you can then correct the computer to get it back on track. And so that’s quite an interesting development because you know, traditionally interfaces have been one step at a time effectively, and David mentioned chaining these things together. We are using a model at the moment called lang chain, which is a bit of a metamodel. And so if you think about the task for instance of, I don’t know, joining two spatial layers together, you can break that down into, hey, I need to find which table represents roads and which table represents populations.

Mike Flaxman (17m 50s):
And then within those tables I need to find the relevant columns and then I need to figure out which of them to join on. And then I need to figure out which attributes are needed from each table. So all of those steps inside of a complex query can be broken out. And that’s a lot of the, the kind of current state-of-the-art work that’s going into, I guess I would say building workflows rather than building single commands. Right. So in our software as we’re we’re developing it, you know, if you ask a specific question about a specific table, it’ll be a hundred percent correct if you ask a question that implies joining two tables together, it might be 80% correct.

Mike Flaxman (18m 33s):
And that’s if there’s a clean join between the tables. And then if you needed to do some manipulation to get a join between the tables, let’s say a spatial operator, it might be 60% accurate. And so as, as David’s saying, it’s also a very interesting time in analytics software because as a software vendor now it’s like how good is good enough? ’cause our traditional user base is used to a hundred percent correct every time. And, and yet you wanna get the advantage of this language understanding and training ’cause it’s really useful without the flip side, without hallucinations or, or lies or or other obnoxious behavior. And we’re at, we’re at a moment where both of those things exist.

Jeff Wood (19m 16s):
That reminds me of a recent conversation we had with Sarah Kaufman at the Rudin Center in New York about autonomous vehicles, which is like at what level will people accept them to be good enough, even if they’re already, let’s say they’re already better than human drivers. Like what is the acceptable amount of collisions that are, you know, allowed to happen before people accept them? And so you’re talking about 80% and it makes me think of that conversation, it’s like, could it be 99.5% where they’re actually accepted or people are still gonna be freaking out about that other 0.5% that actually doesn’t

David Wasserman (19m 46s):
And that depends on the problem too. Yeah. ’cause there are certain problems where if you’re wrong, the stakes are pretty low, but there are other problems where if you’re wrong, somebody dies. Exactly. That’s

Jeff Wood (19m 57s):
Like the autonomous vehicles, right. If say, hits somebody. Right,

David Wasserman (19m 60s):
Right. And so I think like that’s actually been the theme through Michael and i’s multiple of our publications has been this idea of like where you integrate AI is almost just as important of whether or not you do it because there, there are certain types of critical systems where you don’t necessarily want a probabilistic model making the decision, but in some cases if it’s better than human performance then it’s better than human performance. And so it it like, it depends on what the context is, the type of decision and how it’s being made.

Mike Flaxman (20m 32s):
Yeah. So to bring up some planning applications, I mean one of the ones that David and I are bullish on and they’re, they’re kind of two or three themes. I, I guess where we think that these models are already quite useful, but one that’s very impressive and useful is the ability to read enormous volumes of documents. And so as the people that used to have to read, and David I guess still has to read some planning codes and rationalize the difference between, let’s say planning codes in adjacent jurisdictions that don’t make sense together, you know, that’s a great role for these new models is the ability to take all of this massive text and to integrate it and to some be able to summarize it. And so that’s something that even though it still requires checking et cetera, is already useful today.

Mike Flaxman (21m 17s):
And so that’s kind of on the positive side, one of the, one of the more immediate use cases. Yeah,

David Wasserman (21m 22s):
I think a very notable project for me in this realm is the national zoning atlas project, which is trying to map land use across the United States. They’re trying to figure out what is the distribution and look of these regulations on a wide scale because every zoning code is artisanal is what Norman Wright would say. Another planner. One of the things that happens when you have all these codes is that we don’t have an idea of what’s going on. Like we keep on trying to talk about affordable housing, we know it’s a problem, but the insights that the scales that we need to really focus in on, what I think a lot of us might understand to be barriers to development is, is that comprehensive understanding.

David Wasserman (22m 7s):
And they’ve worked with the urban Institute and Cornell’s legal construct lab to leverage natural language processing to try and see if they can extract those codes at a, at a pretty large scale. And I think those are the types of applications Planners should get excited about because it helps us get over some of the barriers that we’ve been facing when so much of how we talk about how our city should be built and designed is so non-standardized and so non-uniform to the extent that it, in order to create a more cohesive picture, it’s impossible. It’s helpful to have somebody that speaks lawyer, right?

David Wasserman (22m 47s):
Right.

Mike Flaxman (22m 49s):
Always. But you can’t afford to pay the lawyer. And I always give the example of Boston, which is 156 or 157, I forget what townships, you know, pretending to be one city and you know, you can’t afford the lawyers to read all 157 code books to figure out why it, it acts as one market even if it’s regulated as 150 something. So that’s just been a perennial issue. Yeah. I do think also I would wanna stress another benefit is, is kind of the ability to look at multimedia data. So things that traditionally, again, computers couldn’t read a diagram. Now they can read diagrams, they couldn’t accept meeting notes that were rough.

Mike Flaxman (23m 29s):
So all of this unstructured data, for instance, public participation data, it’s typically highly unstructured and you know, we’ve never had good solutions as Planners to, to how we rigorously assimilate forms of public input that aren’t convenient to us, frankly. And yet a lot of the feedback you do get from the public is an inconvenient format. And so I think actually that, you know, these tools are a major advance. Dave and I were talking about it earlier. I think for instance they’re multilingual capability chat. G p t speaks 50 languages fluently. And if you look at a, you know, an average large US city, you’ve got 40 plus languages represented.

Mike Flaxman (24m 9s):
That’s always been true. We’ve always tried to, you know, as Planners be somewhat sensitive to that, but nobody can afford 42 plant, you know, 42 translators in a public meeting. So, you know, there’s been just practical impediments to accepting input and providing documentation and, and notice that’s that’s culturally and linguistically appropriate and this is a major advance right on that front. So that’s another thing I think Planners should be jazzed about. It’s not perfect, but it doesn’t need to be perfect ’cause it’s competing with with with doing, you know, English and Spanish, right? Or yeah, it’s typically right. So like it’s 50 languages competing with doing two. But back to the earlier point, yeah. What if the translation is wrong for Farsi?

Mike Flaxman (24m 52s):
It is a problem, right? We still need to get our heads around what level of accuracy is acceptable in what context.

Jeff Wood (24m 58s):
I’m wondering if this much data though can be overwhelming. I mean there’s a lot of places for example, right now that are worried about whether there’s money out there at from the inflation reduction act for all kinds of things from broadband to transportation money. There’s a lot of places that only have like one planner. And so I’m wondering if like this actually might create an overwhelming avalanche of information that that might be just too much.

David Wasserman (25m 20s):
That’s very real. I mean, to some extent our ability to transform data into insight is increasingly becoming a bottleneck. Tools like this can help with that. But at the end of the day, if you’re one planner, is it reasonable to expect them to be able to, like how much, how much people’s expectations change not only of their staff public sector staff, but also generally of each other? And I don’t, I don’t know what the answer to that is either.

Mike Flaxman (25m 50s):
I do see it in general as leveling a bit the playing field there. I mean it, it’s long been the case that communities that are good at grant writing get disproportional resources, right?

Jeff Wood (25m 59s):
True.

Mike Flaxman (25m 59s):
Right. And so this helps a bit, it’s not a slam dunk, but I I think one planner or one grant writer can get a little further with some of this technology, but it may also, you know, we’re at one state in time. So how this affects the overall mechanisms by which grant programs are run and administered is, you know, different questions

David Wasserman (26m 21s):
And they’re using it, the right grants based on some of the surveys that we’ve seen from APA technology Division and conversations I’ve had with M P O directors. It’s, that’s happening. It’s happening. Wow.

Mike Flaxman (26m 34s):
The other slightly nefarious angle on that that David and I have talked about, we, we think it’s kind of, you know, a little bit elephant in the room, so it needs to be addressed. I mean there there is absolutely a, a potential for this technology to overwhelm public comment systems. Yeah. And indeed there’s already some evidence of kind of systematic attempts to do so. So this technology can be used to create fake personal profiles to create fake public comments and

David Wasserman (26m 59s):
Conduct bot attacks.

Mike Flaxman (27m 0s):
Bot attacks. So, you know, all of these things are not exactly news ’cause we, we’ve already been dealing with them, but the level to which they’re occurring, you know, there, there’s a significant threat to at least naive public comment systems that assume that all, you know, that only humans can write. Right? And who knows soon to be, you know, video testimonials too. So the, the ground has changed in terms of public comment and maybe that’s the flip side of what we’re talking about, the ability these models to consume and synthesize public comment in multiple forms. The flip side of that is that they can be used disingenuously to generate large amounts of verbiage, large amounts of public comment.

Mike Flaxman (27m 42s):
And so we need to be on, you know, on the watch for that. That’s something that Planners have had to deal with a little bit, but we’ll have to deal with a lot more.

David Wasserman (27m 50s):
We require them to be more sophisticated about how we conduct online engagement too. Alta planning did actually have a bot attack on one of its public engagement surveys. Now we have a very sophisticated web team, so we were able to sort through that. But you know, not everybody who puts out a SurveyMonkey survey is going to be able to figure out what’s real and what’s not. So I, I think there are questions of capacity that I know APA is thinking about a lot. APA now has multiple memos and reports, foresight reports on the topic of artificial intelligence because there is this recognition that we would be remiss if we ignored some of both the capabilities that the technology offers, but also some of the risks and threats.

Jeff Wood (28m 39s):
Well that’s another thing that I, I had thought about too is that some of the biases too that can be kind of created through this process as well. And, and one of the things we recently chatted about with with U S C professor Jeff Boeing was kind of a machine learned Bias and thinking about Redfin and Zillow and those kind of companies that do real estate listings and how historic inequity is baked into a lot of our systems and processes. And so if that’s learned through a language model or other learning processes, then it’s hard to actually tease it out too in the future.

Mike Flaxman (29m 9s):
Yeah, that’s, that’s another kind of high level highlight for Planners I think because I guess I would say you would be remiss not to assume that a model is biased unless the, the author or the company providing it shows evidence that they’ve tested and found not because yeah, there’s historic Bias of all kinds in these models and the general MO of the models or general requirement of the models is large amounts of data. And so, you know, another example comes from hiring practices, right? So all these resume readers that are hidden behind the scenes are actually mirroring our discrimination patterns from 30 years ago. Exactly. Because they’re trained on historic Datasets. The reason though that, that the companies are using such old historic data is the model requirement for lots of data and there’s only two ways to get lots of data.

Mike Flaxman (30m 0s):
One is it kind of expensive that involves humans curating and the other is just to blast back in time and and scoop up everything. And to point it out, I think there’s a very interesting paper we reference in our APA piece that took a, a completely opposite approach to what the maiden model companies have done. And so they trained the model entirely on I P C C reports about global climate change. And then they evaluated the results of that model with experts and they did find and publish under peer review that indeed the model is much more factually accurate than the open commercial models that are available. And so it shouldn’t really surprise us as Planners because if you think about what went into the two models, one was trained on the entire internet and the evaluation of that model was done outsourced to gig workers and they were given the instructions basically which result is plausible.

Mike Flaxman (30m 58s):
And asking for plausibility for somebody that doesn’t know a topic is kind of an invitation to bad training data. Yeah. But the opposite and the reason why that wasn’t done is the opposite’s expensive to hire experts in any field on any topic is expensive and to get them to evaluate is expensive. And so the commercial incentive to minimize cost and maximize data volume directly contributes to this Bias problem that you’ve mentioned. There are alternatives, however, it’s just that they’re gonna be more expensive by definition. Yeah.

David Wasserman (31m 31s):
And in our memo we talk about two specific papers and a common reframe is that prediction is a mirror where when you think about how we solve problems from the very first step of defining what the problem is to defining what a solution is, to looking at like evaluating whether it’s correct, there’s an analytical frame that you bring to that. And these models, as Mike was saying, are biased by default because of what’s going into, they’re being trained on massive amounts of corpuses from the internet, sometimes unfiltered. And the legal scholars have kind of started catching in on this. One of the papers we’ve cited was Bias in Bias out by Sandra Mason.

David Wasserman (32m 13s):
And it really talks about a lot of these correlative input factors that as it relates to how you use algorithmic designs for criminal justice applications and some of the issues that occur with that. Where in this case, if you think about the problem that you’re trying to solve, which is like how do you identify what are the sources of inequality in, in, in criminal justice applications of algorithms is some of it in the problem definition itself and how you’re thinking about the problem to begin with. I think a good example of this, Jeff, that a lot of people listening to this would be familiar with is the reframe between Accessibility and Mobility, right?

David Wasserman (32m 54s):
Where if we think about what is the goal of transportation? Is it connect people to opportunities or is it to enable pipes that enable people to go as quickly as possible through said pipes? Those are very two different framings of the same problem. And so when we think about Bias, it’s a lot of what’s going into train these models, but it’s also the framing of the problems themselves that people are trying to solve with these models.

Jeff Wood (33m 19s):
I think that’s a really interesting point because we actually, this morning we released an episode with Mike Warren of WSP talking about road user charges and right, the idea of, of politics and you know, policy kept coming up. I kept asking, you know, well could this do this and this do this? And ultimately he was like, well road user charge can do a lot of different things, but it depends on what your goals are and what your policy is related to what you get out of it, right? So you could actually train a road user charge to limit the vehicle size and reduce speeds and all those things, but it’s not gonna happen unless the politics behind it is actually gonna allow it to happen. So, you know, you have this tool whether it’s AI or road user charge, but then you have to set the policy to actually get it, to get what you want out of it.

David Wasserman (34m 5s):
What is it we said Mike, everything’s changed, but a lot of things that stayed the same or something like that.

Mike Flaxman (34m 11s):
Yeah, we’re we’re talking about this the other day that the fundamental hard problems of planning are not wiped away by this new tool, right? The values conflicts, for instance, that have always been kind of core to planning or power asymmetries have not gone away. And in fact, going back to the model training for a second, one of our observations is that the resources that went into training open AI’s chat G P T are just absolutely tremendous and actually disproportional to any effort that has gone on so far on the public sector side. Yeah. So they spent tens of millions of dollars to train million a model with a staff of very expensive experts.

Mike Flaxman (34m 54s):
And there’s the, the largest programs you can think of from National Science Foundation, you know, typically are low millions of dollars for a five-year grant. So there’s a power difference that Planners will be immediately attuned to between who’s building models for what purpose. Yeah. And right now we’re early in this game, so it’s not actually clear whether smaller models will ever be as good. That’s kind of an act of debate technically right now as we speak. My company as well as many others is are, are trying actually to make models that we control that are smaller and more accurate. Exactly for some of the reasons we’ve just been talking about.

Mike Flaxman (35m 35s):
And I work for, for a company that’s gotten a hundred million dollars in funding and sits on a mountain of g p resources, we’re, we’re about as well positioned as you could be for this task. And it’s hard for us. So what does that mean for a county planner or or transportation planning application? So the, the level of resource that’s going into the tools for general purpose is orders of magnitude, you know, three or more words of magnitude larger than what are going into the public versions right now. Now will that be true in two years? We don’t know because we don’t know whether this scale barrier will remain.

Mike Flaxman (36m 17s):
A lot of people are trying to break the scale barrier as we speak. But today, summer of 2023, the best models that private industry can create at a startup, a well-funded startup are nowhere near what the state of the art is. And the public sector is nowhere near even what a startup can do here. So it’s a very strange moment in power asymmetry, but Planners will be familiar with that one.

Jeff Wood (36m 42s):
Anybody. Sounds like the story of history almost unfortunately. I have thoughts also on as Planners we think about long-term horizons and thinking about 20, 30 plans back in 2010 or 2000 is fascinating. What’s changed since 2000 if you’re looking towards those old 2030 plans and now we have 2050 plans, right? Thinking about 30 years in the future as far as the, you know, the NPOs are allowed to go. But we know that things have changed so much since, you know, for example, iPhones coming out in 2009, those types of things happening. And so I’m wondering if you can look into your crystal ball and see into the future what Planners are gonna be up against in crafting these 2050 plans that will look really weird when you look back to on them in 2050.

David Wasserman (37m 29s):
There was a conceptual component to this that comes to mind, Mike, when you and I were talking about this and it kind of comes back to can the can language models be too big paper where they talk about this concept of value lock where a lot of these models are trained once with incremental edits in terms of what goes into them and you could retrain them to update and refresh the information they have. But I think to what you’re talking about Jeff, like the way that we’ve conceived problems previously sometimes gets replicated in some of the outputs of these models. Like going back to that framing problem issue.

David Wasserman (38m 10s):
And so like when people think about using these tools to develop these 2050 plans, I think it’s just something to pay attention to is that like there are very large shifts in values that can occur in communities and sometimes very short amounts of time. Like the fact that, a good example of this is a lot of people are talking about zoning reform in a way that wasn’t being talked about even three years ago. I, I remember presenting on procedural models to academics showing how city engine zoning rules could be used to use G T F S data to show what potential upzoning scenarios look like. And you know, the academics yelled at me and so, and so I kind of backed away from that particular application.

David Wasserman (38m 57s):
But, you know, I think about those types of topics a lot where if we’re not careful with how these models are applied to generate texts for things like plans or grants, like one thing that people want to pay attention to as time goes on is are they framing things in a way that might be akin to how we’ve framed them previously? And are we interfering with social progress in some shape or form by deferring too much to the past as prologue in that way?

Mike Flaxman (39m 27s):
Yeah, so a lot of my career I I worked on simulating sustainable futures and kind of explicitly visualizing long-term, wide area plans. And so I think the positive benefits to visualization of this technology are really substantial. So from that point of view, I think our ability to literally envision long-term futures across large areas, it’s definitely improved, but it’s technology. But as you mentioned, it’s hard to imagine that this social changes that will be brought about by this technology are gonna be so trivial that the 2050 plan is just gonna work out. And so I guess the thing that I expect to happen now that visualization is so much more affordable and especially geospatial visualization is, is so much more affordable is I do expect to see simulations even of longer term scenarios being built incrementally or being able to be visualized incrementally.

Mike Flaxman (40m 23s):
And so there’ll be a lot more spot checking along the way. And I think that having, you know, early milestones will make it easier to see when these plans are going off track. Yeah. But I don’t know what the, you know, public policy response will be. I do think the hardest barrier in that work historically was the difficulty in, in making appropriate visualizations for the community that’s substantially mitigated by this technology. Not perfect, but I think that barrier is, is really important ’cause people do need to see tangible visualizations to think about 2050.

David Wasserman (40m 59s):
And there are very large problems that we’re not a problem historically that a lot of communities are gonna be dealing with. I mean, a an obvious example is climate change where, you know, the last time c o two concentrations have been this way is frankly outside of human history. So when we’re dealing with very large unprecedented situations, I would exercise caution, like if you develop, for example, a Generative model on all the previous historical land use plans as a guide for what you might wanna use into the future, are you replicating things that you want, I guess is the type of question question that a lot of Planners should be asking themselves when they think about these technologies and, and how they’re being used.

David Wasserman (41m 39s):
And I think for me, climate change sticks out as a very obvious kind of scenario that we need to plan for. That requires critical thinking, a lot of engagement and frankly a lot of tough decisions in a lot of communities that a Generative model isn’t necessarily the end all be all, it doesn’t solve all the wicked problems that accompany planning.

Jeff Wood (41m 59s):
It’s interesting because recently there’s been discussion about climate change and flood maps, right? And thinking about going back so far, you know, back to the 1930s, 1940s, those flood maps don’t matter because they’re not reflective of current, you know, trends. And so these models that go all the way back and try to get everything that they can are actually doing a disservice because what’s happening now, this is the coolest summary you’ll have for the rest of your life kind of discussion, right? Because all the stuff that we’re learning now about flood mapping and floods and the intensity of rainfall is so much different than it was back in the fifties, sixties, forties, thirties, all those times. And so taking that data is actually not very useful when you’re making a model for the future because it doesn’t reflect what’s actually happening in the current times.

Jeff Wood (42m 42s):
So it’s, I feel like that’s kind of a, an interesting way to think about and frame those sticky questions that you might have about future planning.

Mike Flaxman (42m 49s):
Yeah, and I’ve just written something on that exact topic because that is a real advantage. So the first Street foundation to give credit where it’s due has, has just issued a, a really comprehensive and interesting report on essentially the complete inadequacy of our national flood program. That’s how I take it as a planner. And

Jeff Wood (43m 8s):
They say they’re coming out in like three years with the right maps, right? It’s

Mike Flaxman (43m 11s):
Like, and know what has a promise in three years that they’ll fix it. Where have you been the last 30 years? But it’s a real dilemma because the pattern of flood that’s shown in those maps, first of all is, is very uneven in, in terms of changes around the country with major pocket in the northeast, including in some economically already challenged communities. So the pattern is complex, the level of change is far beyond what current regulations kind of require people to use. And yes, indeed the current method, when I last looked it up for a study I did in North Carolina, the, the data being used for the baseline was 10 years older than me.

Mike Flaxman (43m 52s):
So I was born in 1964, so they were using 1954 to 1984 data. So it was usually 30 years and that was completely unrepresented of current conditions. And so the law was forcing them to use incorrect information to do infrastructure planning for long lasting infrastructure. That’s a huge problem that the first three foundation report has made very clear. I do think that scenario planning and visualization does absolutely have a role there. And so my optimist tap on that is that now that we have a First Street Foundation map, there’s nothing stopping communities from using that for scenario planning.

Mike Flaxman (44m 33s):
At the minimum there is something stopping them from, from using that in their infrastructure grant. Right? But hopefully if we get a societal habit of being able to do scenario-based planning, we can avoid some of these nasty surprises, including the ones you get from following a 50 year old flood map.

David Wasserman (44m 53s):
It comes back to the importance of clear and accurate information, right. To inform planning decisions. And I I, I think just to pull it back like this was one of the motivating factors for the APA technology Division to write an open letter about Generative AI is that Planners do have an Ethical obligation to provide, it’s like the first rule in the A I C P code of ethics is you, you should provide clear and accurate information to the public as it relates to planning issues. And you know, we talk about different levers where both policies like holding back the use of clear and accurate information, but also though these models can spout incorrect information. I mean, how many lawyers are we gonna see cite cases that don’t exist in the near future because people reach for something like this when they’re on a deadline.

David Wasserman (45m 37s):
And I, I know somewhere there are going to be Planners that might make the same choice as well.

Jeff Wood (45m 43s):
I have to, one last question for you all. We could talk about this all day, I feel like, yes, and for hours and hours and hours, but I wanna kind of wrap it up and give you all the rest of your day. The last question I have is like, what questions do you have about Generative ai, like the potential uses in planning the potential harms, the the biggest question you might have or something that’s unanswered yet something interesting.

Mike Flaxman (46m 3s):
I would go back to David’s last point a little bit. I’m really concerned about truthiness to use a Stephen Colbert expression. So our APA ethics, you know, my alma mater’s motto is all about truthfulness in the way in which we represent data and, and facts and what we bring to the public as Planners. And I’m deeply concerned by the Ethical choices made by some of the companies that have originated these models. I don’t know whether ultimately that will be transcended by other companies offering other models that are substantively more accurate and provide accurate references.

Mike Flaxman (46m 44s):
So when I ask about this in in text circles, it, it’s interesting because it’s, it’s this combination that people haven’t unpacked yet. So the answer is unknown, but if, if you train a model on everything on the internet, you shouldn’t be surprised if you get a lot of spurious results, right?

Jeff Wood (47m 2s):
Everything on the internet, right?

Mike Flaxman (47m 5s):
And so that seems like a suspect choice, but as, as I mentioned earlier, doing the right thing is actually substantially more expensive and these things are already monolithically moonshot expensive to start with. So there’s clearly a countervailing economic at the moment. And yet, you know, as it stands, I wouldn’t use any of these models directly without human intervention exactly for that reason because they, they have been shown to, to confidently lie and to make up references and do stuff that none of us would consider Ethical for a planner to do. But I don’t know fundamentally whether the technical limits or the economic incentives will drive things beyond our control here or whether we’ll be able to, to wrestle the beast.

Mike Flaxman (47m 53s):
And there, there are actually, you know, ways in which public sector and can, can develop models that are suited to the public planning purpose, including the ethics thereof in a, an attractable manner. So that to me is still to be determined.

Jeff Wood (48m 9s):
Our next episode is gonna be on who owns what on the internet too. Oh,

David Wasserman (48m 14s):
That’s a good one. Who

Jeff Wood (48m 15s):
Owns the art that’s being trained? Who o who owns the blog posts? Who’s the style? Yeah, exactly those things. But that’s for next time

David Wasserman (48m 22s):
I think I share, I share Mike’s concerns. We, we outlined them in that article that we recently put out. But I guess on the opportunity side, I think there’s an opportunity for Planners to be informed about how to audit these algorithms. Whether it’s requesting Datasheets to understanding what’s going into them and asking critical questions of vendors that they might work with or talk to that are interested in talking about this technology. But I think when I get really excited about applications of ai, it’s usually tied to the degree, it helps us understand the world as it is today. Whether it’s applications of computer vision to extract sidewalks by companies like Maper or Footpath AI or opia or find crosswalks at scales that typically there isn’t funding to do that type of work.

David Wasserman (49m 12s):
And I, I, I think when I think about these Generative models, there’s kind of an opportunity to marry the regulations that we have in our plans and how we describe them in text with the ability to extract and create a digital understanding of our world that didn’t frankly exist even 10 years ago in a way that can, I think really lead to better and informed planning and more detailed planning in ways and at scales. That was pretty difficult to fathom previously where, you know, if you think about just the, going back to the sidewalk example, the cost of getting a sidewalk inventory in many communities is cost prohibitive. And so people will say, well why is there no sidewalk in my community?

David Wasserman (49m 53s):
It’s like, well you’re lucky if your community sometimes knows where it is. So, you know, I, I think there are applications of these technologies where potentially there’s a way to synthesize, you know, we talking about parsing large amounts of text, our zoning regulations are rules and identify what that looks like at a very large scale, but also how things are today and combine them in a way that might lead to better outcomes. But you know, that’s gonna take active engagement by the profession to make that happen rather than some of the more pernicious potentials that are out there.

Jeff Wood (50m 34s):
Well this has been an awesome discussion and thank you all for coming on the show. I’m gonna put the digitalization link and also your article on AI in the show notes so that folks can read them. I really appreciate that. Where can folks find you all if you wish to be found?

Mike Flaxman (50m 48s):
So I’m just Mike dot Flaxman at Heavy AI or Heavy ai. So easy. Just remember my name and Heavy AI and David.

David Wasserman (50m 58s):
Yeah. So I’ll say I’m at Wasserman Plan on Twitter and David Dash J dash Wasserman on LinkedIn. Awesome.

Jeff Wood (51m 5s):
Well David and Mike, thanks for joining us. We really appreciate your time.

Mike Flaxman and David Wasserman (51m 8s):
Thanks