National Academies Press: OpenBook

Persistent Forecasting of Disruptive Technologies—Report 2 (2010)

Chapter: Transcript of Breakout Sessions for Appendix E

« Previous: Transcript of the Workshop for Appendix D
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

Appendix E
Transcript of Breakout Sessions

GROUP 1

Group 1 Participants:

Moderator: Derek Bothereau

Richard Genik

Fred Lybrand

Jim O’Connor

Peter Schwartz

Paul Twohey

Norman Winarsky

Philip Wong

Michael Zyda

NRC Staff Member: Shannon Thomas

Team Activity: Designing a Scanning System

ZYDA:

The big (loud crash) on the Internet was the Webcast and the fact that somebody had defined HTML. Those are two like boom! events. I remember this very clearly because I had a student come in my mine, Don Brutzman, and he goes, “Mike I got to show you Mosaic.” I said, “Go away, Don, I'm busy.”

SCHWARTZ:

Mosaic was the moment.

ZYDA:

And he goes, “I got to show you Mosaic.” I said, “Don, why?” He goes, “Oh, you'll like it.” So he showed it to me and I'm like – instantly I could – I had a use for it, which was information dissemination. Being a professor at that time I was stuffing envelopes with copies of reprints.

[chuckles]

ZYDA:

All of a sudden I wasn’t in the envelope stuffing business anymore, I just had to build a Web page. And I will tell you within three years – three years later I went to a conference and there were 24 papers in the conference and 23 of those papers referenced papers off my Website. And so to me, the

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

Internet was a big boom! So the question is – but there was a lot of long-term work there.

SCHWARTZ:

One of the places that this diagram begins is decision-makers defining needs. Is a place that this system begins – Recognize we have to have a real client, namely the Department of Defense, Defense Intelligence and so on, do we – It seems to me that one of the important things we have to understand is what is this going to be used for. What do these customers actually want from this information? What they need to know, who needs to know it.

WINARSKY:

These “define needs” is right at up the top. That we surely need.

ZYDA:

So we should have at least one of those.

BOTHEREAU

Peter, you were just posing that.

SCHWARTZ:

Yeah. And it seems to me that we want to begin with a conversation of these --

O’CONNOR:

Who’s paying for this and what do they want?

SCHWARTZ:

We know who’s paying for it but what do they want from it? How do they interact with it? How do they get to participate, what –

[Simultaneous comments]

BOTHEREAU:

Should we start answering that question now?

WINARSKY:

This is what – this openness is what I'm constantly stressed – not stressed in a bad way, just stretching my mind in a good way. It isn't the needs of the funders that will make this a success, it’ll be the needs of people to consume and participate –

SCHWARTZ:

A very good point.

WONG:

I think that’s why I think you can almost – sort of on two streams right? You’ve got one stream which is the government side, which is their user base. And you can look at the other stream which is like the public --

SCHWARTZ:

The participants.

WONG:

The participants, the public. And then think about that openness. Because I don't think the government cares that their interpretation to the system is open but I think the public does.

WINARSKY:

And then they interact with each other.

WONG:

Well, they interact or maybe they don't.

[Simultaneous comments]

Transcripts were not edited.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

And what they want from it may be very different.

WONG:

The systems may even be different.

WINARSKY:

If the open system isn't any use to the government there’s no point in doing it.

SCHWARTZ:

That’s right.

SCHWARTZ:

So it’s got to meet that for sure.

WONG:

Yeah.

SCHWARTZ:

But if it doesn't meet the needs of the participants, they don't participate.

UNKNOWN:

Exactly.

SCHWARTZ:

So I think the point is that there are two different sets of needs, which may not be identical.

WONG:

Right.

BOTHEREA:

Incentives are a huge part of our criteria so I want to bring that in.

WONG:

I was thinking about interaction. I don't know if it’s necessary to somebody on the government side to interact directly with somebody, I guess, on the public side or the participant side, like have a direct communication line.

WINASKY:

No, no, no.

WONG:

But it could be a different type of communication.

WINARSKY:

That’s a benefit. In other words, this open forum about IEDs will have an impact on the government side, for sure.

WONG:

Right. Absolutely, yeah.

WINARSKY:

Right. And, yet, you know, that’s – the contribution shouldn’t be driven by the government. Contributions are driven by everybody that participates whether it’s Facebook++.

O’CONNOR:

We've got some people here who have done more deliverables like this to government entities. What would you think would be an acceptable deliverable from a system like this and what frequency should that be handed off? Is it access to some kind of Website, is it a quarterly report that’s written, what --

GENIK:

I thought the frequency was it’s updated when it’s ready.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Well, yes and no. My experience in this says that actually – with bureaucratic institutions you actually have to have timing and milestones in situations: a quarterly meeting, annual meeting, semi –

[Simultaneous comments]

SCHWARTZ:

The report is delivered at time and the people are prepared to receive it at that time and so on. And those are forcing function. They force you to update, they force people to come to the meeting, to attend and so on.

WONG:

That’s actually a good point, the frequency. Or does the system need to be designed so that if somebody asks a question, which is probably more likely in terms of “Oh, this has popped up on the radar as a potential threat to the country, go ask the smart guys down – the analysts –

SCHWARTZ:

So it’s probably both/and.

[Simultaneous comments]

UNKNOWN:

Yeah, both/and.

UNKNOWN:

Yeah.

WINARSKY:

That’s what I wonder. My mental model has been more like a Disruptipedia, right.

SCHWARTZ:

Disruptipedia, I like that. That’s a good word, Disruptipedia.

[chuckles]

 

SCHWARTZ:

The Disruptipedia system, I like that.

ZYDA:

Who’s checking out the URL?

WINARSKY:

Then it’s always there, people are always contributing.

WONG:

[chuckle] That is a good one.

TWONEY:

Yeah, that’s great.

SCHWARTZ:

Can we agree on a name? I think Norm’s actually come up with a great name, Disruptipedia.

ZYDA:

I like it, I'm in.

TWOHEY:

There’s a subtle point that we've been kind of dancing around which is that I think the assumption here is that the output was going to be the same across all times scales. And it’s not clear that we even want that. So maybe what it is – I've got a question – I've got a quick question so I can get a quick approximate answer, right, versus, “Hey, I need to have a more detailed understanding of these things.” Like these are very different use cases, right,

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

they're even different ways of thinking, like kind of high-level visualizations versus starting at some point and I wander around and maybe I find something that’s interesting. It doesn't have to be the same system that does both, right. It doesn't have to the same results, it doesn't even have to be the same process but just understanding that like these are very different use cases. So maybe what we should do is come up with a bucket sets of use cases and be like “Okay, here’s this kind of thing. This kind of user probably wants this, this kind of user probably wants that,” and we should just pick them off and figure out, “Okay, well, we'll get to some of these guys in the first one and, you know, some of the other guys, well, we won't.”

SCHWARTZ:

A good friend is named Michael Nacht. Michael is the Assistant Secretary of Defense from everything interesting. He does cyberspace, all the cool stuff that’s not main warfare, right. He doesn't have a lot of tanks but he has programmers and all kinds of things like that, rockets, so he does all that stuff. So he asked me a question, he said, “Is there any chance in the next ten, fifteen years that somebody’s going to develop a really low-cost way of getting satellites up – fast.”

ZYDA:

Yes.

SCHWARTZ:

That’s the kind of question that might come along mid-year. A new guy comes in says, “I'm looking at this stuff,” and he says --

BOTHEREAU:

And where would it come from and who’s working on it and –

SCHWARTZ:

Yeah, and how would I answer that question?

ZYDA:

And you want to also invert that question because the question maybe is “When will NASA lose its capability to launch?”

SCHWARTZ:

There’s also that. But the question he specifically – that is not his remit. His remit was to ask the question – Because NASA doesn't launch his satellites, Air Force does. And so the question is really more is there somebody else who could do something --

WINARSKY:

Who could launch –

BOTHEREAU:

Pose a threat.

SCHWARTZ:

A new threat, that’s all.

O’CONNOR:

So there’s the immediate query and then there’s the – If you think about all the kind of – the list, the technology list, there’s also the function of list replacement or list aggregation.

SCHWARTZ:

So am I wrong? Maybe I was wrong to begin with when I said it ought to be a kind of regular cycle, maybe it ought to be question driven. Maybe this whole process ought to be simply – Michael Nacht says, “I got to know does somebody have cheap launchers,” and it feeds it into the system. Maybe it’s purely question driven.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

ZYDA:

So what I envision is something like Facebook where – because if you put that as a post on Facebook then all of the relevant friends that you have could comment and say, “I've got a whole study over here, here’s the link.” Bang.

UNKNOWN:

[chuckle]

O'CONNOR:

I can go post that question on Yahoo right now and in 30 minutes –

[Simultaneous comments]

O'CONNOR:

-- there’s no groundbreaking.

ZYDA:

There’s nothing to build.

UNKNOWN:

That specific of a question.

WONG:

Well, I think you can do that right now but how do you synthesize the information? How do you synthesize it?

SCHWARTZ:

I judge the quality and so on.

WONG:

Yeah, well, how do you actually – like all the stuff we're doing today, right, somebody should be sitting down and synthesizing this into a story or a recommendation, right? So I'm not too sure that piece you can really --

O'CONNOR:

I mean on the existing platforms that are already there there’s one level of synthesis that goes on --

ZYDA:

But then what happens is in this Facebook system we end up with ratings for people. Which is I've got a name, I can click on them, I can go get the info page about them and their expertise.

SCHWARTZ:

So that would – Well, now we do the details on system design. That’s a good example of what we want to include. Rating systems for participants, there’s that kind of system.

ZYDA:

And the value – you know, and there could be a retrospective look at the value that – the posts that person has made in the past. Which is when you see a post from someone with some knowledge you can – You’ve got like the Netflix star system, I click four stars, “This is great,” I click one star, “This guy’s wrong.”

TWOHEY:

You’ve got to be really, really careful about this. When you're rating a movie you're rating a distinct action or you're rating a distinct thing, right. So you're like, okay, before me like this noun had this like overall quality thing. When you start rating a person you really --

ZYDA:

No. Rating the knowledge that they're putting out there in the post, not the person.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

TWOHEY:

Okay. All right.

ZYDA:

And that will lead to overall rating of the person.

TWOHEY:

But you don't want to do that. As an incentive design structure you want to be very careful about rating people

ZYDA:

Sure.

TWOHEY:

Because I can know a lot about computer science and I don't know a freaking thing about combinatorial biology.

ZYDA:

So if you make a comment on combinatorial biology we're going to downgrade you with a low score and you deserve it.

TWOHEY:

Yeah.

WINARSKY:

But the way you're doing it is just what they're talking about not the individual.

ZYDA:

Right, not the person, what they're talking about.

WINARSKY:

But if a person has many times been rewarded with many thumbs up signs from --

ZYDA:

This guy’s a sage.

WINARSKY:

-- in a given area, that person then becomes an influence.

TWOHEY:

Like Bernie Madoff with money?

ZYDA:

No, no, like when you go and buy something on Amazon.

TWOHEY:

The whole point is we're trying to find outliers, right?

WINARSKY:

The unknowns, yeah.

TWOHEY:

If you're trying to find outliers and you're saying, “Oh, great, I'm going to go find all the popular people.”

WINARSKY:

Yeah, you're right.

TWOHEY:

When do the popular kids ever like find the thing that is different, right? That’s the whole point.

TWOHEY:

The whole point we're trying to solve is like not what the popular kids want. So if you go after – [chuckle]

WINARSKY:

True.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

That’s really true. This kid from Afghanistan that’s in a tribe is not going to necessarily be popular but he’s going to come up with something particularly valuable.

UNKNOWN:

Right.

BOTHEREAU:

But how should our system account for some of these kind of lesser-known information sources or kind of some of these – like how do you find them? How do you find those people?

WONG:

Well, I think you have to – can't you rank by relevance. I mean you have to have some system to rank relevance, I mean give certain weights to –

SCHWARTZ:

Yeah, but I think the point is how do you connect in with the outliers? How do we find those people out there, how do we engage them and seduce them into the system –

TWOHEY:

But what question are we answering? What question are we using these people to answer? Let’s stop thinking about how we're going to go rate people and do things. Let’s find like three or four different people who are going to be consuming our system and like how they want to use it. Then once we figure out what we want to give them then let’s figure out how we're going to do it, right. Not like “Oh, yeah, so we're going to go exploit these guys in a village and like make them farm for beans and like it’ll be great for us.”

[chuckles]

 

[Simultaneous comments]

TWOHEY:

I'm trying to say it in the most hyperbolic way possible but –

SCHWARTZ:

It’s good.

BOTHEREAU:

We appreciate that.

GENIK:

I think we should have at least one government customer and one consumer customer and then maybe --

BOTHEREAU:

And I think your template – that’s the guy and that’s the question, right?

GENIK:

Yeah.

GENIK:

Consumer customer can be the movie. Give us the next plot of a Steven Spielberg short that’ll be the next technology to go awry and kill a family or something.

UNKNOWN:

Jim Cameron.

SCHWARTZ:

James Cameron movie.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BOTHEREAU:

As we look back at this as kind of a general flow is there anything that we feel like we haven’t talked about yet? Data feeds I feel like is – We've talked a lot about people, are there data feeds, things that are already out there that could be collected or --

ZYDA:

I have a project right now funded by L&R to build an online game that does behavioral model testing. So we're actually building an entertainment game, putting it up onto the Internet, putting big servers up supporting several thousand people and we're putting various behavioral models that watch play and also interact with the people. But one of the things we're doing in that project is taking in extremes of real-time news feeds. So we're pulling RSS feeds off of CNN.com, off of the BBC, off of economic sites, and plugging that into the world that’s in the game such that it impacts the game play of the autonomous characters.

SCHWARTZ:

Oh, the autonomous characters, not the human characters?

ZYDA:

Well, and human characters by – So, for example, if we see a terrorist bombing coming across the RSS feed, the autonomous characters today are very nervous and upset and angry whereas if they don't see that they calm down over time --

LYBRAND:

That’s pretty cool.

ZYDA:

-- and then they interact with different people. So we're building this, we're at the end of nine months right now, it’s going really well. But the whole idea of news feeds is the big deal because we're going to have this interact in interesting and experimental ways with people.

SCHWARTZ:

So you could answer a question for me.

ZYDA:

Yeah.

SCHWARTZ:

When I think about these data feeds – do I have to necessarily have in a sense a human being reading the data, understanding it, or are there interpretive models for taking it and further digesting it before it gets to me as a human being?

BOTHEREAU:

An automated model.

SCHWARTZ:

Yeah.

ZYDA:

Yeah. Yes, you can do automated stuff.

TWOHEY:

Everyone’s already gone over this. There’s this intelligence apparatus right now for dealing with the things that we know about, right. We already know about RSS feeds. They're already funding people to deal with this, right. Like we don't need to worry – I mean the whole point is coming with the system, it’s going to come with things that we don't know about, right. So I think that –

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Well, no, no, no, no, no. I think that’s not quite right. In other words, the way in which we come up with things which we do not know about may not be out of anomalous data points of themselves but of the interactions of many different things together, the pattern of things which in and of themselves may not be sufficient.

BOTHEREAU:

The correlations not the events.

[Simultaneous comments]

TWOHEY:

But I guess what I was – so there’s – the take that I'd like to put out there is this should be people driven because there’s going – Like for the data to be useful, for people to act on it, for them to disrupt they're going to find it. The axiom that we should us is that they're going to find it before we know about it and they're going to use it and act on it before we know about it. If you want to have an algorithmic solution for finding sets, you want to have a big Google, right, so you want to have a disruption rank or whatever it is –

SCHWARTZ:

But isn't it both?

TWOHEY:

It’s a totally different way of looking at it. Like the algorithmic things that you can do for these kinds of minings, they all assume that people have already built up and structured the data in some way, right? Like the reason page rank works is because everybody on the Internet has a local incentive to point to the thing that they care about and then Google goes and scrapes it and like puts it together. But if what you're trying to do is find the nascent things before they’ve established a pattern, right – So you might have this – you're like building some cloud of possible futures to go look at and maybe head off legislatively or politically or whatever. Then like by definition these things won't have been built, right?

SCHWARTZ:

But it’s a both/and. You're right in some sense but let’s just take a real example. I spend a lot of time thinking about the future of social dynamics in places like the Middle East. So not surprisingly I look at a lot of demographic data, cultural data and so on. Well, in the end, yes, I'm looking for myself, I'm the interpretive engine but before it got to me there were a whole bunch of people and a lot of data that went into massaging the demographic numbers and I'm looking at a bunch of different demographic projections and so on. So there’s an elaborate architecture underneath the human interpretation, some of which is purely mechanical, some of which is simply the integration of a whole bunch of demographic data. So it’s not an either/or, it’s a both/and. In the end it is a human interpretation, I fully agree, but along the way there can be digestion of data that is purely mechanical. That’s all I'm saying.

TWOHEY:

But it just sounds like what you're asking for is better tools to do your existing job. It turns out like the data analysis tools that you have, that they're not as good, they don't have as many feeds. You'd like them to be able to do better, you'd like to do all this kind of analyses but that’s a different question. That’s like “Hey, I've got a pretty good handle on what I want, I've got these six features, can you software tools people like go please

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

make my data analysis things not stink? I'll pay you some money and like make it happen,” right? I don't know, I think this is a different question entirely.

SCHWARTZ:

What’s the different question?

TWOHEY:

The different question is I'm – what – where do we get the data and the leads to start looking at things that we don't even know about yet that should be the problem?

WONG:

Right. I wanted to make on that which is right now – I always get the sense that right now we're thinking about specking out a Website which, you know – for better or for worse, that’s the impression I get, that we're going to create a Website and we're going to have this exchange of information similar to a Facebook. But the reality of it is isn't the information already out there? People are just already using other sites and doing the things that we kind of want to observe?

TWOHEY:

A lot of it, probably.

WONG:

Right. So wouldn't a better solution [be] not to actually create – for us to interface with the end user but for other people who are already – the end user who’s already interfacing with whatever application out there, for those people to look and see what they're doing on those other applications.

[Simultaneous comments]

SCHWARTZ:

Can you give us an example?

WINARSKY:

Mining that information.

WONG:

Yeah, mining that information.

[Simultaneous comments]

WONG:

In an ideal world if you could create a search spider --

WONG:

-- that could go out and collect information from every single Website out there that you're interested in and pull that information and then process it, wouldn't that be as useful as having your own site and being able to process that information?

LYBRAND:

Produce widgets for other people to put on their sites. [chuckle]

ZYDA:

There was a really cool search system that one of my former students has been trying to raise funding on and I liked it a whole lot. I'll describe it to you because it’s cool. I don't think he'll get upset. You're a scientist and you write a technical paper in an area and what you'd like to know is “Did I get all of the prior literature right and is there anybody else working in this area?” You take your PDF and you drop it onto the search engine and it goes out and it reads the whole document and finds people who are working in

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

your area and it tells you those links and it also will tell you in the future when new papers appear on the Net, stuff related to what you're doing.

UNKNOWN:

So it’s persistent.

ZYDA:

It’s persistent. It’s –

SCHWARTZ:

That’s cool.

ZYDA:

And it’s very cool. He’s been trying to raise money, he’s a pretty hot guy so I think he'll get the money. But this is, I think, kind of a core piece of what we're all talking about. It’s different than Google because Google today is what’s there now or what was there when we did our spidering. Whereas I'd love it to tell me --

TWOHEY:

No, no. Have you ever seen Google Alerts?

ZYDA:

Oh, yes, I use Google Alerts but what you really want is – Google Alerts works on just a couple of keywords, this is the whole paper, which is – you know, you're interested in this – the whole thing is the search.

WINARSKY:

The thing that bothers me about this – I mean it doesn't bother me because this is a right direction. So crawling and mining information from other people’s sites doesn't require this very unlikely event that you'll have a very popular site, right?

SCHWARTZ:

Right. Exactly.

WINARSKY:

One in a million get that way. But you do need also these active contributions too. Not just mining other people’s ideas - Because if you have a question like “When will it be low-cost capable to launch your own small satellite?” I'm not sure mining is going to get you that information as much as somebody responding to the question. Literally, you know. I see that question and I'm motivated to answer it.

SCHWARTZ:

Both/and.

WONG:

But do you actually have to host that site to do that or can you use somebody else’s site to do that?

SCHWARTZ:

That’s the question you're asking.

WINARSKY:

Yeah.

UNKNOWN:

Right. I'm just thinking if you were a guerilla right and you don't have a lot of resources you would want to use other people’s assets, right?

WINARSKY:

Right.

WONG:

And that’s what we would want to do, I would assume.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

Right.

SCHWARTZ:

That’s an interesting way to think about it. If you were the bad guys how would you design this system? [chuckle]

WINARSKY:

Right.

BOTHEREAU:

You'd find out about our technology –

WINARSKY:

So as long as you can ask proactive responses to questions and motivate people to answer?

WONG:

But companies do that all the time. You have these community sites that people participate in, companies regularly have their own contributors. They throw up a product idea and they see how the community reacts to it.

ZYDA:

Every single online game that gets released has a fan site where – it’s basically the community site for the game which is bug fix reports and requests for new missions and levels and content. You get incredible smart folks. I think in America’s Army we created a site – we had 150,000 people providing us feedback. We actually selected our several hundred beta testers from the best responders.

TWOHEY:

There’s a company called Wolfire Games. They're actually funding the development of their game by doing this. So they have their fans. They haven’t even released their game yet and they start talking about the process of the thing they're building. So they release like little betas that you get to download if you’ve paid into the system. So they're actually totally funding --

ZYDA:

That’s pretty clever.

SCHWARTZ:

[chuckle]

WINARSKY:

That’s great.

TWOHEY:

Well, because the guy who did it, I guess he made some cult game a while ago so like they believe in this dude and they have this messianic following of this guy and it’s totally working for them.

GENIK:

Going along the lines of somebody asking a question. I don't know if you can address something as abstract as “When will satellite launching be cheap?” The high-level person feeding the input, the customer can ask that question but then that has to be interpreted into “How are we going to do it? Maybe we can make a carbon fiber elevator for example. So you would want to go out on the Web and find people who know stuff about carbon fibers, who know stuff about launching satellites and you'd get the overlap there and that’s something that I don't think exists right now.

TWOHEY:

Don't you just care that it –

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GENIK:

I mean.

TWOHEY:

satellite could be cheap? I mean it’s like I interpreted the goal of this project is to come up with a thing that like “Look, there’s this future that’s coming, maybe not fifteen years away, where launching a satellite isn't that expensive.”

WINARSKY:

Right.

TWOHEY:

Right. And that’s it. At that point we're done because then it’s somebody else’s problem to go like “Look at this” and analyze it in detail and now have the experts study it and figure out – Because now that we've discovered this could happen like great. Now we've hit upon it, the mission of this thing is done. Now we're trying to come up with the next crazy thing where it turns out like I can rewrite my genome –

WINARSKY:

But don't you need – For credibility purposes, for people who are customers to take that seriously aren't they going to say, “That’s great. Did you read that in a Superman cartoon book or do you have evidence that makes me believe strongly in this?”

LYBRAND:

Maybe a system like that is part of a regular report that says, “Okay, gosh, we're kind of persistently looking at all these things that could disrupt. Something like low-cost satellites has a low priority in our last bi-annual update. This time instead of it being a 30-year horizon, it’s a 10-year horizon.” And you balance a system like that that just has kind of a regular survey with a query-based system like what Peter’s friend at SecDef would want.

BOTHEREAU:

One of the things we heard earlier this morning about – what Gilman was saying about what makes a good forecast, was around being able to kind of describe the impact of something like that. What I heard you just say is after you kind of describe the impact there could be kind of a hand-off stage to another entity that would do the tracking or something like that.

GENIK:

Wouldn't we want to know that in advance though?

BOTHEREAU:

Pardon?

GENIK:

We'd want to know that in advance. We'd want to say, “Okay, we know that low-cost satellites are happening, get me a group of experts together and go tell me what’s going to happen once we have low-cost satellites so that we'll be prepared when it happens.

BOTHEREAU:

This is starting to get in kind of to the bottom of this graphic which is allocate resources. So once you seem to have a forecast that’s emerging, how do you actually allocate either people or computer systems to track it and actually start to figure out the details of what’s going on? That’s something we'll want to think about when we do the actual?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WONG:

Are we saying that that is the ultimate output? Like once we've identified that as the output –

BROTHEREA:

I'm asking. That’s my thought I'm deliberately like going on one side so --

WONG:

Well, there's the next layer of the onion that’s --

O’CONNOR:

I think the next layer is the report that would come from the experts might be the output.

WINARSKY:

Right.

TWOHEY:

Or a set of data points that you know you need to be tracking, certain threshold levels.

WONG:

Yeah.

BOTHEREAU:

So when – I think with the satellite that’s kind of a very discrete example. If something happens there there’s going to be a press release or you're going to see something flying up in the air with cloudier issues – picking up chatter on message boards, etc.

WONG:

Well, this is before the press release. This is the idea that – using the launching – an inexpensive way to launch satellites today would be the idea that somebody is gestating somewhere around the Web. That’s what you would want to identify as an output is that.

TWOHEY:

And I think – I don't know, I think it’s pretty clear to everybody – at least to me that like this is going to happen.

WONG:

Right. It’s going to happen. Yeah.

TWOHEY:

Pretty soon, right?

SCHWARTZ:

What?

TWOHEY:

Cheap satellite launches.

SCHWARTZ:

It is. It’s clearly.

TWOHEY:

It’s clearly going to happen.

GENIK:

So somebody should be looking at –

[Simultaneous comments]

ZYDA:

Someone staring up at the sky.

TWOHEY:

I mean it’s already happening, right. Like you already had space experts working on it.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

O’CONNOR:

Well, it depends on what’s cheap.

LYBRAND:

Right.

TWOHEY:

The question is more like “What – “=Like people have been talking about like “Is the cost of orbit going to go down?” This is a question you see, a persistent query, right. I think as long as I've been alive and reading magazines, right, they’ve always said like “Oh, it’s getting cheaper, it’s getting cheaper, it’s getting cheaper.” There’s micro-satellites, you know. So the question is more like what’s something that we're not thinking of yet? Like we're a bunch of white dudes around a table --

ZYDA:

We might not be able to launch satellites –

[Simultaneous comments]

WONG:

And one Asian guy. And one Asian guy.

[chuckles]

 

ZYDA:

We might not be able to launch satellites anymore because we have so much debris in space.

TWOHEY:

Yeah. Or maybe it’s like “Oh, well, you know, this ocean acidification is going to like totally --

WINARSKY:

This what?

TWOHEY:

Ocean acidification totally changes the global carbon cycle.

WINARSKY:

Oh, yeah.

TWOHEY:

Like makes fisheries die off, cause like massive kinds of climate change. So this kind of things happens, right, and entire economies will collapse and this will be a very disruptive – If it happens, right, and people are postulating that there’s some tipping point, right. So maybe the first time you see some paper in some like academic conference about this someone’s like “Oh, well, this kind of matters. Maybe I should –“ Maybe you want to know about this and just – like no one’s sure of what's going to happen but you want to track it a bit more because –

WINARSKY:

Yeah. So there’s two ways to do it. One is a narrative that says – You know an event might happen and it would be a market disruption and you just predict what would make that happen. And then sometimes you follow a technology trend. Like in this case a global warming, carbon dioxide in the air, therefore you read about this ocean stuff. Or another example I was just thinking about – I gave a talk a few years ago on video deception and the ability for processing power in real time to insert or delete objects into video.

TWOHEY:

It’s crazy what you can do.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

So now all of a sudden you could put – somebody who was in a dry country, a glass of wine in his hand as he’s talking on TV and that person’s reputation is destroyed, okay. So that’s a surprise impact that you could never have imagined until you watched the technology curve, right?

GENIK:

Going along with what Fred said, I would want our system – and I've got a, hopefully, short example here. That is let’s say that DIA is looking at detecting deception and they want a system that detects deception using cheap things like a polygraph 100% of the time. Now, of course, we all know it doesn't work. We've done 80 years of work on it and it doesn't work because from these polygraph measurements you can't tell – What you measure is autonomic excitation caused by anxiousness. Now is the anxiousness caused by the person lying or is the anxiousness caused by the person -

WINARSKY:

Situation.

GENIK:

-- just being – situational anxiousness. So that can be sitting down there and persistent. Some paper comes up in a journal on depression that says, “Hey, look, you know when people who have a clinical depression, if you give them a situation which causes anxiousness they have a different autonomic response that changes based upon whether or not they have this disease or not.” So maybe that can give you a clue into getting rid of the --

WINARSKY:

Autonomic response.

GENIK:

When that paper’s published I don't want a flag to come up and say, “Hey, go back and look at all the –“

WINARSKY:

Manic depressives.

GENIK:

Well, “Go back and look at the lie detector stuff because all of a sudden there's some new input there.

WINARSKY:

Either that or recruit the spies out of the category of people.

GENIK:

Yeah.

[chuckles]

BOTHEREAU:

That’s kind of the persistence point though is not reformulating going back.

GENIK:

Yeah. It should sit there and it should keep scanning and if something – now if a question comes up that needed to be answered before then it should alert you that an answer might be out there and at least you’ve got to get your experts back together again because you can't have the same guys working on everything over and over and over again.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BOTHEREAU:

So how often would something like that? I mean obviously you can't predict when there’s going to be a new input that is a game changer but if we're designing a system that needs to have regular intervals or milestones.

GENIK:

Well, it doesn't have to be milliseconds and it shouldn't be years.

BOTHEREAU:

Quite a range.

[chuckles]

ZYDA:

But it might be years. It might be decades.

GENIK:

Oh, for the thing to happen but to sift through the information?

ZYDA:

No, no. But by the time it’s gone to the big jump. The big jump is going to happen because a couple things crash together like Mosaic with HTML.

GENIK:

Right.

ZYDA:

But the underlying technologies were there quite a long time, years, decades. So I don't think – had you gone off and built this system and you had to look at the rich work being done on the Internet, you wouldn't have ever predicted the exponential growth of the Internet. And, in fact, even when Mosaic came out and HTML happened I thought it was really cool just for my neural thought of professor publications. And the fact that Amazon then got created and all of a sudden you could get any book you wanted, that was kind of cool too. It just was the right time. But to try and build a system that could have recognized that that was going to happen ahead of time – not possible, I don't think.

GENIK:

Well, this is where we're having – you're 100% correct but that’s a slightly different system. That’s one that’s trying to predict something that we don't know about and looking into the undefined versus this where we have a customer that’s trying to answer a question that gave us some input, the guy that gave us the money. You can't really say – the only thing it can do is say, “Well, I'm going to go predict what the unknown is.”

LYBRAND:

Right.

TWOHEY:

But it needs to be part of it though.

GENIK:

Yes, yes, yes. It definitely has to --

TWOHEY:

Have you guys ever read Rainbow’s End?

SCHWARTZ:

Brilliant. An absolutely spectacular book.

WINARSKY:

Yeah. Kind of –

TWOHEY:

The whole premise of this book is that like people are going to – Your augmented cognition thing reminded me. The premise of this book is that

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

like people are going to go – in the not too distant future that there’s going to be these like little helpers that go around that help you do data analysis. The number one skill that matters is data analysis. So like your ability to pull in data, analyze data is like what distinguishes you as a human being from other human beings. Part of what I hear is just like “Man, my data analysis tools stink.” Like you want better things for looking at demographic data. You want to look at papers, you want to be able to pull these things in, run these kind of queries, you want just a richer like set of things across all data that’s publicly available and right now like it’s not enough. So it sounds like what you want to do is you want to fund a post-Google search engine strategy, right? Like you want to have a richer, deeper set of things and you want people to innovate on this and you're not even quite sure what you want. So like “Let’s make it happen.” That’s totally than like “Hey, I don't know what’s going to happen. Maybe you could tell me a movie plot.” So like we could –

SCHWARTZ:

But I want both/and. I want a system in which I can interact as a human being, pose queries and observe patterns and be surprised. And I also want a system that is sufficiently broad and rich that is gathering data and presenting it to me even when I'm not asking. I'm the sort of person that read encyclopedias, right, because I don't know what I'm going to find, and almanacs because I don't know what I'm going to find.

TWOHEY:

You must have loved Stumble Upon?

SCHWARTZ:

Yes. But the point is that you don't always know what the query is. The system has to be able to do both. It has to accept queries on the one hand – what’s the low-cost launch strategy, and what am I not thinking about that I ought to be thinking about.

GENIK:

I think the other thing that we haven’t talked about yet is – we've been kind of focusing on the Web app idea and we've kind of forgotten about the idea of using a Delphic method. Maybe we also want to have a system where we collect data from some group like the Jason Group. You fund to send out to a set of conferences or have workshops that try and collect information at that level and bring it in and then discuss it amongst themselves and produce reports.

WINARSKY:

There’s got to be multiple feeds. We can't just have one feed.

GENIK:

Right.

UNKNOWN:

Oh, yeah.

UNKNOWN:

Right.

GENIK:

Right, right. This has a different feed.

WINARSKY:

Gartner could be a feed, right? You know. Wikipedia could be a feed.

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Something we should actually learn from. I'm on the board of what’s called the Research Innovation and Enterprise Council of Singapore, it’s chaired by the prime minister and a bunch of cabinet members and a bunch of outsiders – Clay Christensen, the head of R&D of Siemens, Novartis, Dupont, the president of Stanford --

GENIK:

Is Judith what’s-her-name on there? Judith Swain?

SCHWARTZ:

No. But underneath us there are about four hundred scientists organized in about thirty or forty different committees and different disciplines, different arenas and so on and they're doing a whole bunch of stuff feeding us and then we're taking it and then finally making decisions about program allocation.

ZYDA:

So they're kind of doing what I was talking about.

SCHWARTZ:

A billion dollars here, a billion dollars there and so on.

GENIK:

Next thing you know it’s real money.

SCHWARTZ:

Well, it’s twelve billion a year. It's already real money.

[chuckles]

 

ZYDA:

That’s Singapore dollars though, right?

SCHWARTZ:

Twelve billion Sing. Yeah, okay, so that’s only about eight billion U.S.

TWOHEY:

I'll take that. You want to give it to me?

SCHWARTZ:

You'll take that. It’s bigger than NSF.

[Simultaneous comments]

O’CONNOR:

The fx rate is in their favor, right

SCHWARTZ:

Yeah, right now. My only point is that it is a fairly elaborate system of human beings on the one hand massaging and feeding. And it’s quite large scale, I mean it’s hundreds of scientists in these kind of panels and Delphi groups. Paul Saffo is on one of them.

ZYDA:

Again, this is the mechanical turf, which is humans feed – intelligence feeding upper-level stuff.

SCHWARTZ:

Yeah, that’s right. Exactly.

TWOHEY:

It was interesting. When I told a bunch of friends that I was coming to this thing and they're like “Oh, wow.” So I asked a bunch of grad student friends like what they thought was going to go change the world. It turns out if I just

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

asked them like “Hey, what do you think’s going to be like the coolest thing that’s going to happen?” I got a lot of really interesting responses.

SCHWARTZ:

Like?

TWOHEY:

And then – hold on. But then I asked a different question, I said, “Can you tell me about disruptive – “I asked people “Can you tell me about the things that are going to be disruptive technology?” And they all kind of looked at me like I was on Mars, right.

ZYDA:

What planet are you from?

TWOHEY:

Yeah.

ZYDA:

It’s a bad term.

TWOHEY:

It’s like “What?” So the whole way you frame the question really matters. So all my friends, the two themes were robots – they think that like small robots with intelligence are go around and they're going to do lots of things -

LYBRAND:

Remotely

[Simultaneous comments]

SCHWARTZ:

Or spybots

GENIK:

I've got a video of that with me.

TWOHEY:

Then the one they thought was the sequencing technology in genome

SCHWARTZ:

Genome sequencing.

TWOHEY:

Like the massive – like the combination between like massive gene sequencing, synthetic biology and computational stuff, that it’s going to totally change like the way we looked at medicine and the building blocks to do DNA.

WINARSKY:

Yeah, so –

TWOHEY:

I got that just by asking people and I think that there’s actually a lot of value in asking – So the NSF, right, they have a bunch of money and so they fund a bunch of graduate students, right. So if you look at the people that are actually implementing most of the technological change, right, not IEDs but like if you look at technology-driven disruption, right, it’s almost always coming out of an American university funded by U.S. taxpayer dollars. So as long as you're going that you might as well ask the graduate students every semester to put in three paragraphs: What’s the coolest thing that’s happening in your area – like your direct area? What’s the coolest thing that’s happening in your field? And what’s the coolest thing that’s

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

happening in science, right. Just like literally three paragraphs, less than five hundred words each. How many people do we fund, right? We probably fund –

SCHWARTZ:

Make it a requirement of every NSF grant. [chuckle]

[chuckles]

 

O’CONNOR:

Log in to Disruptipedia.

[laughter]

 

SCHWARTZ:

And you don't get your check until you’ve cleared off the Disruptipedia.

O’CONNOR:

Fill out your security number with your hard-copy submitted application.

[Simultaneous comments]

ZYDA:

And the grad students that did the work are from China.

SCHWARTZ:

I think your point’s well taken. I think it’s a good idea.

[Simultaneous comments]

TWOHEY:

What I mean is if you want to do some kind of Delphic method I think that you're going to get – So I was thinking – one of the questions I wrote down was – there’s this question of experts versus crowd, right. How are experts made? How did you get to be an expert? Well, usually it’s passion plus time. So you're sitting there just like doing…

[Simultaneous comments]

TWOHEY:

But if you're really into like movies or you're really into biology. Like my friends that are in genomeology, they’ve been looking at slides since they were like seven. That’s the kind of people that you want to be answering these questions. So maybe you should just – Because we already have – If you want an expert in biology, right, you call somebody up and you’ve already got it.

WINARSKY:

By the way, you don't always call a graduate student up. It could be these crazy people that are not in …

[Simultaneous comments]

SCHWARTZ:

But that’s an “and” it’s not an “or.”

TWOHEY:

I'm not saying you should only do this.

WINARSKY:

Yeah. Yeah.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

But I think it – one of the ideas we had early on was to create a system that basically looked at what people were doing for their Ph.D. dissertations and say what are they making personal commitments to by the choices that they make in their lives as a way of mapping potential disruption. We found that actually difficult to try and pull off but it basically gets at the same point. I think it is (a) an interesting source when you have the graduate students who are right at the leading edge of each of the disciplines.

WINARSKY:

But here’s the problem. The things that we're looking for are the 1% or one-tenth of 1% events which most great universities would look at somebody who might try and want to do some – who was really excited by some cool idea like cold fusion or something like that and say, “What are you thinking of writing a dissertation on?”

O'CONNOR:

But I think one thing to take notice of is that if you have these grad students who are these experts, self-declared experts, in one place that is kind of a honey pot for those fringe people.

WINARSKY:

Right.

O'CONNOR:

They’ll be attracted to that community. But it is figuring out, you know –

TWOHEY:

Then you can data mine to your heart’s content with the Yahoo submit to the…

WINARSKY:

But imagine if I'm in the MBA school and I said –

TWOHEY:

Well, you're not going to [..?..]

WINARSKY:

-- two years ago and I said, “I'd like to write a master’s thesis on what would happen if subprime mortgage--”Well, that isn't so strange, is it? What would happen if --

O’CONNOR:

It was four years ago.

[Simultaneous comments]

SCHWARTZ:

If one of the big investment houses vanished in a weekend.

WINARSKY:

Yeah.

[chuckles]

 

WINARSKY:

You might not get your degree is what I'm getting at.

SCHWARTZ:

That can't happen.

[chuckles]

 

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

TWOHEY:

So there’s this question of like what are the axioms of the system. You wouldn't get your degree, so maybe there’s some value in just having explicit – making explicit what the assumptions are.

WINARSKY:

Right.

TWOHEY:

Then you go play what-if games every quarter with your experts. Just “What if this thing didn't hold?” My mechanical engineering friends say that like when they're trying to build something they go and they play this game about like what-if. What if they could just like kind of disable one of the laws of thermodynamics or –

[Simultaneous comments]

SCHWARTZ:

Aren't we saying – this goes back to, I think, your point that you began with, Rich, of a system of systems. A question I have is this. We're identifying a number of different strategies for surfacing things, for judging them and so on, whether it is a Delphi-like approach, whether it’s talk to the graduate students, whether it’s looking at Ph.D. dissertation, whatever it may be. So I think what we want is a variety of such strategies and then a meta strategy for learning from those so that they get better. We drop things that don't work, we add things that do work and that we are building a kind of learning dynamic over time that says, “The first pass through this, Version 1, we tried five different strategies and four of the five worked but three didn't yield must useful –“ or “One didn't yield much useful. We're going to keep four and we're going to add another one in the next generation,” and so on. Is that the kind of model that we have?

WINARSKY:

Yeah.

UNKNOWN:

Yeah.

UNKNOWN:

Yeah.

LYBRAND:

Well, I have a proposal – let’s be the first group to go out and play with the table.

UNKNOWN:

Right.

UNKNOWN:

Right.

LYBRAND:

I mean I think everybody has the one or two points that they really care about.

[Simultaneous comments]

WONG:

Just before we go though, we talked about the technology – well, we talked about the sort of idea of PhDs and graduate students and thinking about the cutting edge technology, right, and I just wanted to sort of bring us back a little bit to, well, actually, use cases, right, and to the guy again in sub

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

Sahara Africa, right, and he’s not necessarily using the latest nanotube technology –

SCHWARTZ:

A very good point, right.

WONG:

That’s the use case that I'm most interested in.

WINARSKY:

Yeah.

SCHWARTZ:

So how do we get at that one?

WONG:

Huh?

SCHWARTZ:

How do we get at him?

ZYDA:

Buy him an iPhone.

WINARSKY:

You start from his –

SCHWARTZ:

What?

ZYDA:

Buy him an iPhone.

TALMAGE:

Actually, they're already on cell phone technology there. They didn't put landlines in, they went straight to cell.

WINARSKY:

You start from his need. I want one person to kill a thousand – or

SCHWARTZ:

But how do you find out what kind of things that they – How do you find the guy who’s figuring out how to put an IED together?

[Simultaneous comments]

UNKNOWN:

That could be some of the global --

TWOHEY:

The question’s actually irrelevant, right, because like we now know that IEDs exist. That’s a totally separate question

[Simultaneous comments]

O’CONNOR:

They did this before. I mean there were booby traps before and re-labeling them

TWOHEY:

But what I mean is after the first month of IED attacks in Iraq like this whole question of disruptive technology was moot from like our perspective here because at that point in time it’s a disruption that the military’s already noticed it happening and …

SCHWARTZ:

I'm sorry. I have to disagree. If before the invasion of Iraq in 2003 – if a year and a half before they said, “You know, one of the biggest threats we're going to face is IEDs and here’s why.” “Well, you know, the vehicles we're

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

sending over are wrong, the armor we're sending over, body armor, is wrong and our guys aren't trained and we don't have detectors. So what we really need is redesign of our vehicles before we go in, we need some new detectors and we need some new training.” That did not happen and a whole lot of people died as a result of that.

WONG:

Right.

SCHWARTZ:

So it is not accurate to say this is an old technology. It wasn’t in the perception.

[Simultaneous comments]

TWOHEY:

No, no, no, what I said was different.

ZYDA:

There's some social things in the military that are pretty critical that you're going to have a hard time getting around because if you – In the military there are two different camps that do this job. There’s an analysis community and there’s a modeling and simulation community and, you realize, those are two completely separate communities. And if you go to the analysis community, they’re ops research guys, they’ve got some mathematical modeling behind and they love to do big statistical runs on big engagements. They don't do much in the way of thinking of surprise, they don't do much of anything in thinking “How do we go off and look at the behavioral end of things?” So I think that’s going to be the – that’s, to me, one of the fundamentals that you'll have. So if you were going to go and improve the analytical capability of the military you would actually smash those two communities together and cross-train them and provide a tool that could get them out of their comfort with their current toolset.

BOTHEREAU:

So to Fred’s suggestion, I think we probably should go over there and kind of start laying it out, okay. Is everybody okay with that?

[relocate to project table]

[resume conversation at table]

TWOHEY:

Like if you look at disruptive technology in the Silicon Valley area it’s – VC funds have a ten-year time horizon, right, and they have a 70% failure rate, like 10% like massive success rate. It’s only asset class. Where if you bought into the average of the asset class you'd lose money.

WINARSKY:

Exactly.

O’CONNOR:

For venture capital.

TWOHEY:

Yeah, venture capital. So you need to be aware of the fact that like what we're trying to do is we're playing venture capital on ideas.

WINARSKY:

Right. Very clever.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

TWOHEY:

Like just the model we have for success and how you evaluate these things – we need to have – Whatever system you put in place, right, you can't say “Oh, it didn't find this thing in this one year,” right, or “It didn't find this one disruptive technology so let’s chuck it right away.” Like you need to have a – The cool thing about government, the only cool thing about government is it has enough money and time and patience to have a long time horizon, right. That is the key resource

LYBRAND:

Now you sound like a college endowment officer, that’s – it’s – yeah, yeah, yeah.

WINARSKY:

Uhhhhhhh.

TWOHEY:

But what I mean is like it’s the key thing, right. Because you can be like “Okay, I can wait, right, I can be patient.” So like this is cool, right, so this lets you --

GENIK:

Well, it depends what part of the government you're talking about.

TWOHEY:

But I mean as long as you're not spending ridiculous sums of money doing it – like if over ten years it only costs you a million bucks, right, and like the upside is huge, right, then it’s okay. Right. Right.

WINARSKY:

So your point is --

GENIK:

That has to be leveraged somehow because you're not going to get – over ten years if you got one person and $100,000, you're just paying his salary. It’s got to be leveraged with some other kind of funds.

TWOHEY:

Of course. Of course, right. But I guess what I'm saying is like it’s okay.

GENIK:

I don't mean to shoot down your idea.

TWOHEY:

No, no, no. No, but I mean like it’s okay, right, like if we come up with a portfolio of really leveraged ideas and it’s like “Oh, yeah, so most of these things will fail and we're okay with it,” and like some politician doesn't make his re-election bid by like taking your thing and like gunning for it and knocking it down. [chuckle]

GENIK:

Like DSSC.

TWOHEY:

Well, there’s some prediction market, right, for terrorist things and…

WINARSKY:

Right.

LYBRAND:

Yeah, yeah. We had the people who did it come in and spoke to us --

WINARSKY:

I wasn’t there then.

LYBRAND:

-- before we got the smart guys involved.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

[chuckle]

TWOHEY:

What I mean is like somebody got re-elected because they're like “Oh, this is a really stupid thing.

GENIK:

Yeah, because I shot this program down because – not because it was necessarily bad but I can show that it was bad to –

SCHWARTZ:

But this might be input, that might be output.

[Simultaneous comments]

TWOHEY:

People far more intelligent and like with much better social skills than myself [chuckle] need to be involved in selling --

GENIK:

Yeah, don't put the scientists in front of --

LYBRAND:

That’s all right. The guy who has to deal with manufacturing problems, I always start at the end.

SCHWARTZ:

Do we want to assign any meaning to the geometry?

TWOHEY:

Go for it.

LYBRAND:

I just started writing.

THOMAS:

So rectangles are output – or at least --

WINARSKY:

Rectangles are output –

O’CONNOR:

What are they normally?

WINARSKY:

Triangles are --

THOMAS:

Whatever you want it to be.

[Simultaneous comments]

TWOHEY:

So what do you consider if you're a customer?

LYBRAND:

Well, I would either come up and say, “Where’s my regular report that I want every X.” I don't know what time period, you tell me. Or I would say, “Here’s my question. Microstats, tell me more. What’s the –“

WONG:

I was just thinking about this. The idea of having a regular report puts us back in the paradigm of like what we've done historically. You expect like “Well, this threat is not going to come up for another two years.” Well, guess what, things don't necessarily take two years or our prediction timeframe might not be two years. What if it comes up a year in advance, what happens if it comes up in six months? Right?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GENIK:

No, no, you're right. You're right.

LYBRAND:

Based on the sponsor feedback I just don't see how they --

THOMAS:

Well, if you are the analysts --

GENIK:

Well, without at least a yearly conference or something.

THOMAS:

…team that's going to work it and then you have the customers who are higher ups. The analysts themselves might want to bring them a quarterly update.

LYBRAND:

This might not be external.

[Simultaneous comments]

SCHWARTZ:

So lets have three things here. Output #1 is query response. I got a question, I ask the system, I get a response. Second is analyst-driven output. I see something interesting going on somewhere in the system, we've surfaced something. “Customer, you didn't ask and we're not at a regular report interval but a thing has come up you need to know about.” So analyst-driven output. And then regular systematic output because the truth is that bureaucratic systems need that.

GENIK

So that’s a third, yeah.

SCHWARTZ:

That’s a third.

[Simultaneous comments]

WINARSKY:

I'm struggling with this regular stuff.

SCHWARTZ:

It won't survive. It won't endure.

[Simultaneous comments]

WINARSKY:

Regular reports. Let me tell you what my problem is.

[Simultaneous comments]

GENIK:

It might be as needed sometimes and it never happens.

WINARSKY:

My problem is we're talking about disruptive events that are unpredictable.

SCHWARTZ:

Right.

[Simultaneous comments]

[microphone noise]

SCHWARTZ:

I'm saying three different forms of output.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

Okay.

SCHWARTZ:

One is driven by the event, okay. So I discover something interesting in the system and you're my customer. I call you up, “Norm, hey listen, we found something interesting you need to know about.” That’s one. That takes care of the case that you're just talking about.

GENIK:

That’s analyst driven.

SCHWARTZ:

That’s analyst driven.

WINARSKY:

Can we follow that a little bit?

SCHWARTZ:

Okay.

WINARSKY:

So would somebody – taking the previous conversation, somebody said, “I just saw Mosaic working on the Web.

SCHWARTZ:

Right. Good example.

WINARSKY:

I mean working on ….

SCHWARTZ:

I just visited University of Illinois and I just saw the first version of a browser and I think this is a really big thing.

[Simultaneous comments]

WINARSKY:

I think this is a big thing, okay. So this is sort of a scanning, proactive surfacing of a big thing which has low probability of having happened, some surprise event.

SCHWARTZ:

Correct. That this is really going to take off, a network effect could happen.

WINARSKY:

So that’s like you saw it. Then there’s the predicting of those things that might happen. Predicting Mosaic is another whole level of –

TWOHEY:

Because you look at the Xanadu, right, or whatever project – There were a couple of people that were trying to make hypertext systems before Mosaic, right, and they –

WINARSKY:

Right.

TWOHEY:

And they – “How do I know about this? It’s like they didn't – like they didn't ….

SCHWARTZ:

We didn't have the means to do Xanadu when Ted Nelson first came up with it. Now we've done it! It’s not exactly Xanadu but it’s close.

BOTHEREAU:

Have we decided on what we want the shapes to signify?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

No, we have not.

[Simultaneous comments]

LYBRAND:

I'm still running with squares as output. Squares I mean rectangles.

[Simultaneous comments]

SCHWARTZ:

Analysis and decision.

WINARSKY:

Analysis and decision.

[Simultaneous comments]

SCHWARTZ:

These are outputs, inputs, analysis and decisions.

TWONEY:

Where are we starting? Are we starting –

SCHWARTZ:

Well, we started here with outputs.

[Simultaneous comments]

WINARSKY:

This is the inputs at that side of the table.

SCHWARTZ:

Left hand is input, this is output.

[Simultaneous comments]

LYBRAND:

So our manufacturing guy looked around and said, “You start with what you want out of the system.” So that’s my bias.

[Simultaneous comments]

SCHWARTZ:

And we've got three classes of outputs now.

[Simultaneous comments]

SCHWARTZ:

We've got output that is query driven, output that’s regular system driven, and output that is analyst driven.

[Simultaneous comments]

LYBRAND:

And I'll tell you what, if anyone wants to add further classes, if there’s a fourth and fifth or other derivatives or subclasses who will that –

[Simultaneous comments]

O’CONNOR:

So this is the end of the system and then we'll go back and build the rest.

LYBRAND:

Right. Right.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

There’s outputs that comes from people participating in whether it’s a game or whatever. It’s neither query driven nor --

WONG:

Like what’s the user-facing output?

LYBRAND:

This is user-facing output.

WINARSKY:

It’s kind of like I'm using Disruptipedia and that’s my output.

WONG:

This is the government sort of –

WINARSKY:

Right?

WONG:

But what about the guy who’s actually participating?

WINARSKY:

Yeah, there’s somebody --

LYBRAND:

So there’s intermediate-stage outputs. There’s components. I mean it’s the supply chain that generates the --

WONG:

It isn't intermediary because to that person it’s an end output.

WINARSKY:

Yeah.

SCHWARTZ:

Well, okay, but I think that’s a good point. There’s a fourth output which is continuous -

LYBRAND:

Could be raw data.

SCHWARTZ:

Well, it could be continuous output --

LYBRAND:

Continuous update.

SCHWARTZ:

-- in the form of Disruptipedia display.

WINARSKY:

Yeah, exactly.

LYBRAND:

Oh, very cool. That is really cool.

WINARSKY:

Let me just point out, the reason that this is so important is, again, we said “open.” So the output isn't just to the analyst.

SCHWARTZ:

Exactly right. Exactly right. Very important.

[Simultaneous comments]

BOTHEREAU:

Is it an “ipedia”?

LYBRAND

And Paul was saying that the URL is available if you want to get it.

UNKNOWN:

[laughter]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

What? Disruptipedia.

UNKNOWN:

We should get it, right?

BOTHEREAU:

No, I was defaulting. You guys made the Internet, not me.

WINARSKY:

We all worked together. [chuckle] Disruptipedia.

SCHWARTZ:

Disruptipedia. We've got it.

WINARSKY:

You got it. Go buy it. Go to GoDaddy.com.

[Simultaneous comments]

[chuckles]

 

BOTHEREAU:

No, no, that was Paul’s idea.

[Simultaneous comments]

SCHWARTZ:

That was Norman’s idea. He gets the credit.

[Simultaneous comments]

WINARSKY:

I was following your idea.

ZYDA:

InYourFacebook.

SCHWARTZ:

InYourFacebook.

[laughter]

 

THOMAS:

This is all going down in the transcript.

[laughter]

 

LYBRAND:

Fred Lybrand. InYourFacebook.com.

[laughter]

 

WINARSKY:

Before this goes public somebody register that domain.

BOTHEREAU:

We have our outputs and the build back into kind of the steps. It looks like squares are outputs.

SCHWARTZ:

Right.

WINARSKY:

Domain –

TWOHEY:

Squares are government outputs, right?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Why don't we go to the other end and what are the inputs?

UNKNOWN:

Yeah. I'll keep writing this down and come join you guys.

[Simultaneous comments]

SCHWARTZ:

So we're going to use these guys down here for inputs.

LYBRAND:

So systematic – So these are like that – support the bureaucracy [..?..]

[Simultaneous comments]

WINARSKY:

This is query now. It’s not just a query, it’s the out --

LYBRAND:

The query to system and response. So it’s a micro-satellite.

[microphone noise]

SCHWARTZ:

We've actually done that systematically and what we find is it is extremely rare to have the science-fiction writer who has the right level of perception for it. Vernor Vinge is one of them actually who does. In fact we had Werner at one of the committee meetings for just that reason.

TWOHEY:

I would have loved it. Sorry, fanboy.

[laughter]

 

SCHWARTZ:

Vernor’s a …

[Simultaneous comments]

SCHWARTZ:

By contrast, Bruce Sterling is wonderful and Bill Gibson is not.

ZYDA:

No, he’s boring.

SCHWARTZ:

Yes.

WINARSKY:

You're on tape –

SCHWARTZ:

No, that’s okay.

WINARSKY:

This is a public transcript –

[laughter]

 

[Simultaneous comments]

SCHWARTZ:

Bill is a good friend.

ZYDA:

After Burning Chrome and –

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

He’s a great writer, he’s just not a great participator in these sort of things.

ZYDA:

His recent writing’s horrible.

WONG:

I feel like such a nerd. [laughter]

SCHWARTZ:

But Sterling is great in these situations and so is Werner.

TWOHEY:

What I mean is there’s this – We've been focusing so heavily on the technical side and like – But there’s also this – A lot of these innovations don't come from people with technical backgrounds, right.

WINARSKY:

Exactly.

TWOHEY:

So if you're like “Hey, look –“ If you want an idea for an artist or something to mine to come up with something there’s this – Here’s these people that are postulating on the future, like go look at Futurepedia for some potential things. Feel free to pull out of this.

WINARSKY:

Right.

TWOHEY:

And then if you gave people the ability – pretend we're building this – I'm leaping like six steps ahead, right, but pretend we built this Website and then you have people put these predictions out there and they're able to go attach supporting pieces of information to these things every time they happened, right. So like every time something happens it starts more to confirm your prediction and then you get points or notoriety or whatever it is. Then that might give somebody else some idea that comes with it. I don't know. I mean just getting people that don't think like me is really, really important because --

ZYDA:

So, you know, this sci-fi writers thing was tried by the USC Institute for Creative Technologies.

[Simultaneous comments]

WONG:

You don't go to the technology experts but you go to the people that are – like if you could get people that are actually thinking about – like the low-tech guy, right.

TWONEY:

Yeah.

WONG:

Because 9/11 was low tech.

TWOHEY:

9/11 was really low tech.

WONG:

Really low tech, right.

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WONG:

I mean they obviously had access to like flight manuals and things like that, right, to teach themselves how to fly.

[Simultaneous conversations]

TWOHEY:

Tom Clancy predicted it.

WONG:

That’s true, yeah. [chuckle]

[Simultaneous comments]

WINARSKY:

Another kind of input we said in our previous conversation was the narrators.

UNKNOWN:

narrate stuff.

WINARSKY:

The people who narrate –

TWOHEY:

After 9/11, a couple months after, my friends and I got together and we realized –

WINARSKY:

-- a story

[Simultaneous comments]

TWOHEY:

The terrorists were stupid. They were particularly uncreative in what they were doing.

[Simultaneous comments]

WINARSKY:

This is public content period.

[Simultaneous comments]

UNKNOWN:

Journals, tech papers --

UNKNOWN:

Tech papers, that's a good one.

WINARSKY:

Also everything that Google can find.

UNKNOWN:

Exactly.

ZYDA:

The interesting thing about tech papers on the Internet … The stuff before that is still not there…

[two conversations carried on simultaneously]

BOTHEREAU:

We're trying to figure out what does this start with. So we're going to lay out a V.1 system, what’s going in.

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

TWOHEY:

So who do we have that uses the Internet that goes to lots of like really crazy places. It’s actually – you want people who travel. When I want the skinny on some crazy country --

WONG:

Yeah, travel channel travel blog.

TWOHEY:

I go ask my friends that travel like what’s – “So what’s really going on in like Tanzania.” So I asked my friends that were in the Peace Corps and these other things because they have this very different perspective and –

SCHWARTZ:

You're right. As an ex-PCV I'm on the blogs and the intelligence that you get – it’s actually fascinating. When I went in the Peace Corps before we got on the airplane they told us, “Don't talk to anybody from the embassy, they will be CIA trying to subvert you.”

[chuckles]

 

SCHWARTZ:

That was literally what we were told before we get on the plane to go to Ghana.

[Simultaneous comments]

SCHWARTZ:

Within an hour of landing there was a guy next to me saying, “So, welcome to Ghana. We'd love to talk to you. Which village are you going to?” They were right there trying to subvert the Peace Corps.

[Simultaneous comments]

TWOHEY:

I have a friend who is a redheaded Peace Corps volunteer.

[Simultaneous comments]

WONG:

NGOs, all of those guys.

[Simultaneous comments]

GENIK:

Out there.

GENIK:

That's fed from the output.

UNKNOWN:

This is data collection on kind of technologies that could be out there.

TWOHEY

No, it's perspectives; perspectives. Because I think if you --

SCHWARTZ:

What did you mean, are these narratives?

WINARSKY:

What I meant here was I meant something that is saying, “Here’s a –“

SCHWARTZ:

It’s a story I'm interested in.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

Tell me how bioweapons could be used against us in New York City in four years. That’s a narrative.

TWOHEY:

Like lines at the airport for passports suck a lot. It’s terrible. So if you gave somebody the opportunity to cut the line by just telling a five-minute narrative of their trip and some cool things and then you could even have some automatic transcription of this, right.

BOTHEREAU:

If we were to label this section kind of like a narrative --

O’CONNOR:

Inputs.

UNKNOWN:

Don't they ask you where you’ve been anyway?

[Simultaneous comments]

WINARSKY:

There's narrative inputs and then at the end there’s going to be another narrative.

SCHWARTZ:

Narrative inputs.

WINARSKY:

Narrative output.

[Simultaneous comments]

WINARSKY:

And then there’s going to be narrative outputs.

[Simultaneous comments]

THOMAS:

May I make an unreasonable request? If there’s any way that this could stay as much as possible one conversation. I know that’s a really silly thing to ask but at the same time we have a transcriptionist who’s struggling very hard to keep up with everyone.

FSPKR:

Sorry. [chuckle]

UNKNOWN:

That’s great. Thank you.

WINARSKY:

I'll tell you what, here’s how you do it. This is the baton. Whoever holds this can speak.

[chuckles]

 

UNKNOWN:

We have a football around.

[Simultaneous comments]

[chuckle]

 

SCHWARTZ:

Well, here’s a question. How do we get – I don't remember who posed the question about the guy who’s trying to build the device in the desert. He’s

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

not a grad student, he’s not on a blog, he’s not publishing a scientific journal.

[Simultaneous comments]

SCHWARTZ:

What’s the process for not necessarily engaging him but seeing what he’s up to. How do we find out what he’s doing?

ZYDA:

He’s unimportant --

WINARSKY:

I think that’s good.

ZYDA:

-- unless we actually have someone on the ground who’s listening to him or talking to him.

[Simultaneous comments]

SCHWARTZ:

NGOs and aid workers?

[Simultaneous comments]

ZYDA:

We should get that down because how are we – who’s doing the listening?

LYBRAND:

The journalists.

SCHWARTZ:

Exactly.

SCHWARTZ:

But is this the category that solves your problem, Mike? So we've got NGOs and aid workers. So we have some guy out there --

ZYDA:

No, a Special Forces guys.

SCHWARTZ:

Okay, we add them, Special Forces guys.

GENIK:

It could be just – Well, I wouldn't put them on the same plain as military.

SCHWARTZ:

It may be different. These are input sources at the moment. How we get the input may be different. So it’s basically people on the ground in remote places that we're aiming for.

ZYDA:

That’s right.

SCHWARTZ:

So that’s the category.

ZYDA:

Boots on the ground.

GENIK:

Or tennis shoes or sandals.

LYBRAND:

or shoes.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

What we're trying to do is to enable people on the ground to input to this system in some way.

[Simultaneous comments]

TWOHEY:

What about also the actual people themselves? Because all those are first derivative on the person. That’s some U.S. guy who’s going and talking to them and reporting back. What about just – what if you had a random citizenship lottery. So like we gave a hundred people a U.S. citizenship and to get it they had to go enter some information in here and you went and talked to them. You'd get flooded with --

ZYDA:

Nonsense.

GENIK:

Yeah.

TWOHEY:

But you'd have the data that you could actually start looking at.

GENIK:

Well, look at police tip lines when you offer a big reward.

[Simultaneous comments]

ZYDA:

There’s a $25 million-dollar reward for Osama bin Laden. Are there any tips? No.

BOTHEREAU:

So this is great. I kind of clustered these together because they seem like a parallel –

[Simultaneous comments]

BOTHEREAU:

This is an information gathering exercise so –

SCHWARTZ:

This is the front end.

BOTHEREAU:

We can add to this but where – If we were to get this information where does it –

SCHWARTZ:

That’s the next step. We're still at this step.

BOTHEREAU:

We're still here, okay.

[Simultaneous comments]

WINARSKY:

Actually, that feeds over here.

[Simultaneous comments]

SCHWARTZ:

This is something else.

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

No, no, this was an input. This was the idea that somebody back up there was telling a story. I want to know the answer to that.

BOTHEREAU:

Ah, okay.

SCHWARTZ:

In other words, the story is there’s a bunch of guys out in the desert doing XY and Z and I need to look for that.

BOTHEREAU:

Yeah.

SCHWARTZ:

That’s actually part of the query.

WINARSKY:

But we also create narratives by having looked at this data.

SCHWARTZ:

Yes, that’s right. That’s another output.

GENIK:

That can also be an intermediate step.

WINARSKY:

I'm getting confused between inputs and outputs.

SCHWARTZ:

Yeah, it’s circular.

GENIK:

I'd like to make suggestion that this is enough input for now and we should look into process. We can always add to this.

BOTHEREAU:

Do people feel comfortable with that?

GENIK:

Because we only have like an hour.

SCHWARTZ:

Half an hour.

[Simultaneous comments]

SCHWARTZ:

Okay, so we're gathering all this information, what are we doing with it?

ZYDA:

Google.

SCHWARTZ:

Now we got some triangles here.

[Simultaneous comments]

WONG:

Well, is the question next how do we gather that information?

GENIK:

Well, I think it’s dependent upon the information that it is. I mean this you have to have people talk to them.

BOTHEREAU:

Before lunch there was some talk about incentive systems for getting people to participate. Trying to get people to partner with you. Are there some incentive systems here that we need to keep in mind that are going to help us process and –

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Yeah. Are we just simply assuming that there’s somebody in an office somewhere looking at public journals, calling journalists, reading the blogs, talking to people, going out there, what’s happening? How’s that one working?

WINARSKY:

Actually, we were also going to do a scraping, a – the Web scraping of --

BOTHEREAU:

Yeah, let’s get that.

WONG:

It just feels like the solution is some kind of like super spider thing that -

TWOHEY:

You know, there’s a company called Eighty[?] Legs and you're like – you can outsource your Web scraping.

ZYDA:

Why bother like doing it yourself?

SCHWARTZ:

We haven’t gotten that far quite yet but that might be an example, outsource. Yeah, that can be outsource.

WINARSKY:

We just need that as input no matter what.

SCHWARTZ:

But what do we imagine is happening with all this stuff?

BOTHEREAU:

Right. So where does it go? Is it going onto a Website, onto a computer system, are there –

SCHWARTZ:

Twelve people sitting in a room reading it every week? What’s happening?

BOTHEREAU:

Yeah, are people reading it? Where’s it going?

TWOHEY:

Why are we assuming there's just one system?

WONG:

Yeah.

SCHWARTZ:

Or systems.

LYBRAND:

We're not.

SCHWARTZ:

We're not.

WINARSKY:

No, many. Yeah. Oh, you mean many [..?..]

BOTHEREAU:

It’s a V.1 flow but it could have --

WONG:

Well, put it this way. Maybe the first thing here is that where do people in this group put their information right now. As opposed to trying to get them to put it in the place that we want them to put it, where are they putting it right now?

WINARSKY:

Newspapers, journals, conferences, blogs.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WONG:

Because that’s exactly where we --

[Simultaneous comments]

SCHWARTZ:

But it isn't being synthesized and coherently gathered.

WONG:

Right.

SCHWARTZ:

We have in the Monitor Group something we call the Discontinuity Committee, which I chair. The objective of that committee is to do three things. Is to (a) identify discontinuities by looking at all these kinds of things, (b) actually effectively engage decision-makers but (c) get better at doing it. Study how we do it and improve it. I would like to propose that in here somewhere we have the equivalent of such a group called the Disruptive Technologies Committee and its job is to interface with the output and users, to constantly learn and refine the input and then constantly improve the methodologies of analysis.

WINARSKY:

Yeah, that's good. Very good. Great.

SCHWARTZ:

So that up here at the peak of this we have the Disruptive Technologies Committee.

LYBRAND:

Should we think of this like a company?

WINARSKY:

Yeah.

LYBRAND:

With a board, an advisory board.

TWOHEY:

….this idea before. It’s like what we really want is like a disruptive – We're not trying to build just one product or being in one market, right, we're trying to provide a perspective to a bunch of other people, right. So they're going to give us some amount of – whatever this thing they're going to design, right, they're going to get some amount of money, they have a long time horizon and they just want to make sure that like the world didn't blow up on their watch, right, and like we kind of kept the train on the tracks.

LYBRAND:

I think the model of this is similar to a VC in that it’s a prospecting engine and it’s always turning over every rock and it’s very good –

SCHWARTZ:

That’s not a bad analogy, actually.

LYBRAND:

I think it needs to be more corporate or other – again, in structure just because those partnerships are always so personnel driven and this needs to be – for it to persist over decades and decades.

SCHWARTZ:

So what do we do on those committees? I'm on a – So we all go out and go to conferences, to workshops, laboratories, constantly gathering information. We come back in, we listen to presentations from companies and so on.

LYBRAND:

Yeah, yeah, yeah.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

We debate it and discuss it. That’s the committee I'm talking about, okay.

WONG

Right.

TWOHEY:

What I mean is it sounds like there should be something that’s like – there’s going to be continually new ways to analyze this data, right. Like it would be hubristic to assume that like we're going to come up with the ultimate be all, end all today or this week or whatever. It should just be like, okay, well, if you do something and it takes a million dollars and it runs two years and then – say it takes $10 million and it runs a decade, right, and you do this ten times, you do this once a year for a decade, right. If those things come back with – that’s actually not that much money in the government’s scheme of things, $10 million a year, right. Like if you're able to diversify risk across a bunch of different things then that’s probably worthwhile.

WINARSKY:

True.

TWOHEY:

If you get a couple – so what’s the value – Let’s put some money on this, right. If we go and run this movie contest – which I think is actually a pretty awesome idea, you know --

SCHWARTZ:

A script contest.

TWOHEY:

Come up with these like cool terrorist scenarios for our next action movie, right, for the next G.I. Joe movie or whatever it is --

ZYDA:

But, you know, the cool ideas are not going to be on the Internet. If somebody does turn in a really cool video --

BOTHEREAU:

Especially if they can cash it in for – with our committee.

TWOHEY:

But what I mean is if you're the place where they're going to cash it in then you get the idea. And then you win.

WINARSKY:

Right.

ZYDA:

But nobody's going to see your video.

TWOHEY:

No, no.

ZYDA:

Because it’s instantly classified.

TWONEY:

Well, but that’s – so it’s

ZYDA:

I'm just --

LYBRAND:

Suspend that for a second though – you know, we can play with that, come back to the security thing.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

TWOHEY:

Like just don't classify it. Like call it fiction and then like everyone will laugh at it and say it’s not going to happen. Or put a little delay on it, put some – You know, someone’s like “Oh, we're ready to liquid explosives on the airplane.” And so you go, “Okay, fine, no more drinking.” Whatever it is.

BOTHEREAU:

That seems like it’s a bit of input but it’s also a bit of a process

WINARSKY:

It’s a motivation.

WONG:

But why wouldn't you just take it off the studio sites, like just figure out we get it from them directly. They're pretty good at incentivizing people to bring the scripts, the stories, so --

BOTHEREAU:

Yeah, I mean it could be another way. That could be one way –

LYBRAND:

It could also be its own site.

[Simultaneous comments]

WONG:

Sorry.

LYBRAND:

It doesn't need to be its own thing. To your point of look for where this information already lies.

WONG:

Yeah.

TWOHEY:

Look, the entertainment industry is always trying to like wheedle Congress for certain things, right. So if they attach a rider

[Simultaneous comments]

GENIK:

This is the analysis section. This is analysis.

TWOHEY:

Like you need a little bit of soft power kids.

[Simultaneous comments]

WINARSKY:

Delphic groups?

SCHWARTZ:

Yeah.

ZYDA:

That’s what they do.

WONG:

That's a good idea.

BOTHEREAU:

So how else are we processing?

SCHWARTZ:

Well, do we imagine – There’s two classes of processing: human processing and machine processing. Does machine processing play any role in taking

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

any of this or is this really all going into human beings and the human beings are interpreting it?

WONG:

Well, I think part of it’s quantitative and qualitative, I guess. There’s going to be quantitative and qualitative information that’s coming out of this. So qualitative, I think you're stuck with human beings, on the qualitative side?

TWOHEY:

But, for example, I can go write a Web crawler that goes and crawls all your known terrorist Websites, rips out all the first and second-level information from that, applies statistical machine translation and does topic classification on this, right.

WINARSKY:

Right.

TWOHEY:

You're only going to have like a 20% accuracy at the end of all those steps because you know the Arabic to English thing, the text categorization, these are still emerging technologies, which doesn't mean it’s not worth the government’s energy like funding research in an area but you could mitigate some of your risk by doing this and so why not? Right? I would actually be kind of shocked in the secret – the high side stuff in the Defense Intelligence Agency doesn't do something like that that I don't know about right now. I'd be disappointed in the taxpayer dollars.

WINARSKY:

The beauty of what you're saying is automated processing might only work 20% of the time but actually the truth is these signals keep coming. These signals don't go away, especially if they're going to become a disruption. So even if it worked only 20% of the time, over time that kind of processing could surfaces this kind of input.

[Simultaneous comments]

SCHWARTZ:

Again, if in that 20% you only get one significant hit every two or three years it’s still worth it.

WINARSKY:

Right. Well, but who determines if it’s significant or not.

SCHWARTZ:

In hindsight. In other words, you saw something and it turned out to be important, that’s all.

WINARSKY:

Okay. So what’s success though? I mean if –

TWOHEY:

It’s a counter-factual problem, right. Like you didn't get cancer because you didn't smoke so is this a success for, you know, what?

WINARSKY:

Yeah.

TWOHEY:

It’s hard to measure the lack of bad things, right, as a --

BOTHEREAU:

So out of this are we trying to maybe come up – Let’s say out of this we came up with 50 to 75 technologies we want to track.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Hundreds.

BOTHEREAU:

Hundreds maybe. How many would we want to allow to push into the rest of our flow? I don't know if we have time for all of them or the research for all of them so how would we – would there be a smaller set that – would there be a kind of filtering that would happen at some point?

[Simultaneous comments]

WINARSKY:

Well, that’s what Paul was saying.

SCHWARTZ:

We're trying to get at the filtering strategy at the moment. How you take the hundreds and turn them into the half a dozen that you really want to focus on.

UNKNOWN:

Yes.

TWOHEY:

Our take on this is that you can't do it accurately. Like that’s actually our fundamental axiom so –

[Simultaneous comments]

SCHWARTZ:

Yeah, accuracy is not a –

TWOHEY:

I'm not going to sit here and tell you, “Yeah, you guys can totally do it accurately.” Right.

WINARSKY:

Right.

ZYDA:

So we don't need –

WONG:

I just thought we missed one.

ZYDA:

-- the bottom point there which is really good data.

GENIK:

There's always space at the top.

TWOHEY:

Like I was telling you, I don't know if anything is good

SCHWARTZ:

What do you mean by that?

WONG:

It means that these are all things that people are doing intrinsically. We could stimulate discussion within the community if there’s a question we want to ask.

GENIK:

You can hold a workshop.

WONG:

Right.

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GENIK:

Like when the Army puts out an RFA, they invite everybody and only the nuts show up.

WINARSKY:

Who was talking about use cases before?

GENIK:

You’ve been to one of these. [chuckle]

WINARSKY:

I'm getting stuck on a use case and how this would work. Let’s walk through one use case, okay, with a launch of a small --

BOTHEREAU:

Great.

SCHWARTZ:

Cheap launch capability.

GENIK:

So we've got a connection from there to here then.

SCHWARTZ:

Yeah, so back here the loop is I have a question. I'm Assistant Secretary of Defense and I say, “I want to know about whether in the next five years somebody’s going to develop a cheap launch.”

GENIK:

Where’s our question.

WINARSKY:

Okay.

WONG:

Or would it be like something “I'm thinking about invading another Third World country and I'm going to secure policing action around that country. What do I think about over the next couple of years as I prepare to enter that country?”

SCHWARTZ:

That might be another use case. It’s a different one.

WINARSKY:

So let’s just follow one and go through it.

[Simultaneous comments]

WINARSKY:

I'm trying to figure out – I don't what the connections are, do we all agree.

BOTHEREAU:

I just took a survey of everyone else’s, we are clearly winning.

[laughter]

 

TWOHEY:

Nice.

[laughter]

 

SCHWARTZ:

So taking your point, Norm. Take a look at this, what would happen. Michael Mack calls up the director of this program and says, “You know, I got to know about this.”

WINARSKY:

Yeah.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

So he sets in motion with his analytical group – he’s got a group of people. Here are all of your information resources. Different people within that group will tackle different pieces of this different set of sources.

WINARSKY:

Right.

SCHWARTZ:

So I'm going to read over the output of Web crawlers, I'm going to get –

WINARSKY:

Who’s “I” now?

SCHWARTZ:

A member of this committee up here. We've got our committee and Michael Mack, the Assistant Secretary, has called the head of the committee and said, “I want to know about cheap launch.”

WINARSKY:

So that committee is an analyst committee, is that right?

SCHWARTZ:

Yes.

WINARSKY:

Is that what we're doing?

BOTHEREAU:

Should that move down?

TWOHEY:

Let’s just make another one.

WINARSKY:

That moves over here in the analysis stage of –

[Simultaneous comments]

BOTHEREAU:

Peter, that just made it sound like that’s kind of the start. You get the call and that happens.

LYBRAND:

Yeah, that use case is the one that works through.

WINARSKY:

This is our analysis object.

SCHWARTZ:

Okay.

WINARSKY:

So here.

GENIK:

I mean we start with something raw that somebody says and then we do something with – we do analysis on it.

SCHWARTZ:

I'm calling – the reason – this group does a bit more than just simply. In other words, the call comes in, “I want to know about low-cost launch.” Now five of us get assigned to that problem of a committee of twelve and we are given some –

WINARSKY:

Per output.

WONG:

I think the first thing you would do if you were asked a question like that would be you'd first postulate with a hypothesis for why low-cost launch

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

would be possible, right? Like do we think that there’s a new technology development? Do we think there’s a Third World country that’s copied something? Blah-blah-blah.

SCHWARTZ:

We think there’s an urgent need that many countries have therefore they're trying to solve the problem.

WONG:

Right. But the first thing you would do is not necessarily jump into the research side. The first thing you would do would be to break that sort of question down either into sub-questions or hypotheses, right?

BOTHEREAU:

That’s great.

SCHWARTZ:

Cheap propulsion, cheap navigation.

WONG:

Exactly. Right.

GENIK:

So there’s an analysis --

WONG:

What are the drivers of that going to be?

[Simultaneous comments]

TWOHEY:

Maybe something about breaking down the question?

SCHWARTZ:

Into operationalizable questions.

WONG:

But I think it also has to be breaking down into hypotheses because you really want to have a story, right.

WINARSKY:

Right. You have to have the evidence.

WONG:

We have to go back with a story, right?

SCHWARTZ:

Yeah, the narrative in this case is Country X – Hugo Chavez will develop satellite-launching capacity –

WINARSKY:

Exactly. Now what’s our analysis stage?

GENIK:

Well, yeah. But that analysis of the question has to happen in the system, it doesn't happen outside the system. So what I'm saying – I'm making a point that this line is correct, it goes right here.

WINARSKY:

It goes to a narrative. Then we're --

GENIK:

And then here to here. Here to some –

WINARSKY:

What is that triangle? What is our analysis?

GENIK:

Are we writing our analysis?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Analysis.

GENIK:

Breakdown.

SCHWARTZ:

Break down the question into sub-questions and then feed it through the various input sources.

WONG:

But then I think you have to develop – you have to develop a --

GENIK:

These are the inputs too.

WONG:

I think you have to develop a hypothesis.

SCHWARTZ:

The question goes back out to this.

UNKNOWN:

Develop a hypothesis, yeah.

SCHWARTZ:

So which blogs are likely to surface it? Which -

LYBRAND:

Story.

SCHWARTZ:

Science-fiction writers surveys, a way we gather.

WONG:

Otherwise you'll gather the ocean.

LYBRAND:

I may be misusing the shape but to Philip’s point, if I got this query wouldn't I want to have some kind of dashboard that I went to that was consistently surveying this?

SCHWARTZ:

Ah, good point. Yeah, yeah, that might be.

LYBRAND:

And it might throw me out something that said, “Number of mentions across all these things –“

GENIK:

You're right. That’s an output that’s kind of things

LYBRAND:

And it’s just inherent –

WINARSKY:

That’s this.

LYBRAND:

Oh, okay. Yeah, yeah. Oh, cool. Yeah, yeah. And then –

SCHWARTZ:

It could be. Not necessarily but it’s one way it could be done, yeah.

[Simultaneous comments]

LYBRAND:

Low-cost launch is a function of these however many inputs.

SCHWARTZ:

Inputs. Propulsion, fuels, guidance systems.

GENIK:

There’s still some intellectual content in there that you are breaking down.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

[Simultaneous comments]

TWOHEY:

They make equipment for mining like fuel, oil.

WONG:

Do you want to put that actually in

[Simultaneous comments]

TWOHEY:

And they actually have internal systems for answering these kind of questions

GENIK:

I thought that this guy. This is just the output going all the way back here.

TWOHEY:

How do you break it down into actionable items?

WONG:

You have to break down that question into sub-questions.

TWOHEY:

Like Fortune 100 companies already have this issue like right now.

[Simultaneous comments]

WONG:

Then you have to develop hypotheses for the story because at the end of the day you're going to have to tell a story back to that guy, right?

[Simultaneous comments]

GENIK:

You're right, you have to separate this. I think they go down here after the narrative input.

[Simultaneous comments]

BOTHEREAU:

Okay, so we've got our inputs. We've got our inputs and then we were talking about kind of the processing strategy.

[Simultaneous comments]

WINARSKY:

Here's processing right here.

GENIK:

So now we're breaking down the question.

BOTHEREAU:

So this is – this is – you’ve got narratives that are – this comes out of the narratives.

GENIK:

The narratives are an input to breaking down the question.

WINARSKY:

So let’s keep following it.

GENIK:

Relevant hypotheses.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

TWOHEY:

Can I ask a silly question here? Just a fundamental assumption. Who’s doing this? Is this all like Defense Department contractor people?

GENIK:

No. We just say the process.

TWOHEY:

Okay, I'm just saying.

SCHWARTZ:

Here are arrows, by the way.

[Simultaneous comments]

BOTHEREAU:

So that’s a good question. We're actually going to in the next stage of this after the break talk about the human and technical requirements specifically, are there any government and non-government partnerships, that kind of thing. For now I think we should just try and get like a skeleton mapped out and then we'll say where would you need to outsource.

WINARSKY:

Well, I could imagine – yeah, it could be a company, it could be the Disruptipedia that --

GENIK:

Make it out of macaroni too.

UNKNOWN:

[chuckle]

GENIK:

Brings me back to kindergarten.

WINARSKY:

-- you're just providing input and people around the world are providing output.

BOTHEREAU:

Now.

WINARSKY:

Okay. So automated processing brings us to these --

UNKNOWN:

Yeah.

UNKNOWN:

So do we have the incentives that we talked about?

SCHWARTZ:

We have not built in the incentives.

BOTHEREAU:

Before the break we had a rich conversation about incentives.

WINARSKY:

Right.

BOTHEREAU:

How can we layer that in? Because that seems like a differentiator for our system, I felt.

WINARSKY:

Peter’s going to start movies.

BOTHEREAU:

Incentives.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

TWOHEY:

I think that’s a really awesome – I think sort of – What really worries me about the way we're designing this thing is that we're designing one process, right.

WINARSKY:

Right.

TWOHEY:

And I think that like the fundamental way to do this is actually to have a bunch of totally distinct processes with groups of people that don't even talk to each other.

SCHWARTZ:

Well, that’s what we've got here.

TWOHEY:

No, but what I'm – Look what we're doing, right. We have one arrow, we have one set of things that are going through, right. Like where would we put this --

BOTHEREAU:

So how would you suggest – That’s a good point. How would you suggest changing the process.

WONG:

Well, I think it’s still a single process but you're saying that it’s more than one, right?

TWOHEY:

The way I think of it, I think that there’s like three separate systems that you have kind of running in the little Olympic-style rings.

GENIK:

Oh, out of the narratives?

TWOHEY:

You have one set of people that are looking for the next Xanadu, right. So I'm going to use the Web as the example here.

[microphone noise]

TWOHEY:

If somebody says, “Oh, there’s this really cool idea it'd be nice if it was built.” So like what – here’s a horizon, here’s a possible future maybe this should be around, right. And then you have another set of things looking for actually actionable technically made things. So when you see Mosaic you're like “Oh, my gosh, this is really cool.” And then you have some automated tools that help you tie this to this notion of Xanadu or whatever it was. And so all the other analysis and the contingency planning that you did for when that happened, now that kicks into effect and that comes in. And at the same time when you're looking at the intersection of those two things you have another group of people that’s like “Oh, hey. We're going to look at the possible futures and what’s happening right now and we're going to come up with a bunch of different movies, a bunch of different narratives.” And then you feed these to the decision-makers so then they can ask questions that you then feed back. So that’s the way I think of it. It’s not as this one monolithic process that just kind of lumbers along.

WINARSKY:

That’s true.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GENIK:

We started this exercise by saying let’s trace one thing through. That’s why we ended up with one.

TWOHEY:

Yeah, but what I'm saying is we've got to be really careful because it’s not just --

GENIK:

Right. Right, right.

BOTHEREAU:

So how could we capture what you just said on the table?

GENIK:

Is that arrows from here all going in – I mean we don't want to make this a giant nexis.

WONG:

Or would you say that it’s just – are you saying it’s the same process or a different process here?

TWOHEY:

I think you have like three distinct processes that are kind of running in cycles that just interact at various points.

WONG:

But you just like duplicate it like so there’s a multiple, right. You're doing this in parallel.

TWOHEY:

But I mean – okay, that’s just my thought. Like please disagree with me.

WINARSKY:

No, it’s right.

WONG:

No, I think it's right.

GENIK:

No, it's right. It's right we discussed it.

GENIK:

It’s how you do transcription in India, right? You have four guys that you're paying three bucks an hour, right, and then you cross-check the --

WINARSKY:

Right.

SCHWARTZ:

The incentive question is an interesting one. The kinds of people who we want to input this are smart, knowledgeable, observant people. Why would they wish to participate? Well, it could be money. That’s not likely to be it. But it does strike me that one of the reasons that people participate is that interesting people are listening. That is they believe that somebody at the other end is going to be paying attention that they would like to be heard by. So hypothetically, let’s put it this way, an incentive where if we constructed that committee properly, i.e. that the right twelve people – that all the people we would like to have participating wanted to be heard by. It’s Steve Jobs, it’s Steven Spielberg. I don't know, it’s Barack Obama.

ZYDA:

Uma Thurman.

SCHWARTZ:

Uma Thurman. Yeah, better yet.

TWOHEY:

So you do this, right and you're going to go to Davos or wherever it is.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Wherever, yeah. But in other words there’s some – you are heard by the right people as a powerful incentive.

GENIK:

But, Peter, I think that that’s part of our process.

UNKNOWN:

That’s great.

GENIK:

Not part of the input customer.

SCHWARTZ:

But this is how you get people to participate.

GENIK:

Right. Right.

SCHWARTZ:

It’s an incentive.

GENIK:

So we should make some stuff --

BOTHEREAU:

Should we make an incentives --

SCHWARTZ:

Attractive committee.

BOTHEREAU:

Like maybe an incentives box and list out – you want to list out some of the incentives so we get that stuff.

SCHWARTZ:

Yeah, you know, you want to be there. People want to be in Davos because everybody else is there they want to be heard by.

WINARSKY:

That’s true.

SCHWARTZ:

I go to Davos every year because it is really an interesting place to be, right, and everybody else is there.

BOTHEREAU:

So let’s get some of those incentives.

WINARSKY:

So there’s another thing about analysis that I'm puzzled by – not puzzled but torn by. We crowd source a lot of the inputs. We're not crowd sourcing the analysis.

UNKNOWN:

Um-hm. [yes] That’s a good thought.

WINARSKY:

Why don't we --

SCHWARTZ:

Yeah, we need multiple analysis.

WINARSKY:

Why don't we use the crowd to analyze. If you're asking how to launch micro-satellites, why don't you ask the crowd how to launch them?

SCHWARTZ:

Good. So we need some feedback out to the crowd and back again.

WINARSKY:

Yeah. Yeah.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GENIK:

Isn't that then a two-way thing there?

SCHWARTZ:

Yes.

WINARSKY:

Yeah.

BOTHEREAU:

All right. Let’s make it.

[Simultaneous comments]

WINARSKY:

So we have the crowd-sourced analysis. And that goes back – that’s all this stuff.

GENIK:

That’s in parallel with --

UNKNOWN:

Move these over.

WINARSKY:

It doesn't just have to come from that. The crowd can analyze any of these questions if you get them motivated to. Right. Now you really need motivation.

BOTHEREAU:

Crowd source analysis. And what comes out – so the crowd is – kind of all these inputs are feeding.

WINARSKY:

All this stuff can eventually be inputs –

SCHWARTZ:

And through the Disruptipedia they get to participate in crowd source analysis.

WINARSKY:

The crowd will make up – You want to tell me how to blow up a ship in New York harbor. Send it out to the crowd, these ideas, and let them [..?..]

SCHWARTZ:

So we need an arrow from the Disruptipedia over to here.

BOTHEREAU:

Where’s the Disruptipedia, down here? Okay. We need another arrow.

GENIK:

Down to –

SCHWARTZ:

Crowd source analysis.

WINARSKY:

Right.

UNKNOWN:

I'm also interested in –

ZYDA:

Not much process in the middle.

UNKNOWN:

Yeah, that’s right.

UNKNOWN:

It's an Etch-a-Sketch.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

ZYDA:

It’s an agile process.

[chuckles]

 

UNKNOWN:

They do everything in pairs.

UNKNOWN:

[chuckle]

BOTHEREAU:

Fred, where do these –

LYBRAND:

I was just throwing those out as if I – If somebody came to me with a query, I would hope I would have some sort of answer where I would have built some kind of analysis.

WINARSKY:

That’s just breakdown.

BOTHEREAU:

It’s kind of GUI almost stuff, right?

LYBRAND:

Right. I mean it’s --

BOTHEREAU:

So I might put it more --

LYBRAND:

Similar things are recorded elsewhere over the annals of this committee.

WINARSKY:

Right.

BOTHEREAU:

Is it all right if I put it more toward the end?

LYBRAND:

Wherever you would like it, including the circular file in the corner.

WINARSKY:

So now we have somebody’s output that they want, which is a micro-satellite query as to when it will or is it possible that it will be done. We go to the millions of inputs, we analyze that potential and we out put the results. So what am I missing.

[ongoing side conversation between Wong and Twohey]

O'CONNOR:

Wouldn't the Disruptipedia have to be up here?

WINARSKY:

Yeah, it is kind of up there.

GENIK:

Disruptipedia is an output.

BOTHEREAU:

Maybe this should be a part of it up here too.

SCHWARTZ:

[laughter]

GENIK:

Well, you know, our problem is that this is a line and we want it to be a circle. [chuckle]

[continuing simultaneous conversation with Twohey and Wong]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

That’s kind of this, crowd source input.

GENIK:

Mmm –

WINARSKY:

Disruptipedia? Yeah?

BOTHEREAU:

So a system and that collects all the inputs.

GENIK:

I think what we mean by here is almost individuals though, isn't it?

WINARSKY:

Yeah, this is individuals providing --

GENIK:

This is individual provided and this is system provided.

BOTHEREAU:

So we've got the inputs, we've got crowd source, incentives.

WINARSKY:

Right.

SCHWARTZ:

We've got one incentive, that’s all

BOTHEREAU:

Yeah

SCHWARTZ:

We need some more incentives.

BOTHEREAU:

We need some more incentives. We had some other things here about --

GENIK:

Script writing --

UNKNOWN:

Maybe access to unique people. You were kind of mentioning that.

SCHWARTZ:

That was the one I had!

BOTHEREAU:

Interesting people, script –

WINARSKY:

Participation in a script.

UNKNOWN:

Yeah. Reputation.

WINARSKY:

Reputation.

UNKNOWN:

currency or virtual currency –

WINARSKY:

Yeah, you could do a mechanical turf kind of thing.

[Simultaneous comments]

GOLDHAMMER:

Ours ends in world peace, where does yours guys’ end.

[laughter]

 

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ:

Disruptipedia. Oh, good.

[Simultaneous comments]

LYBRAND:

World domination.

[laughter]

 

GOLDHAMMER:

World domination, excellent.

[Simultaneous comments]

WINARSKY:

There’s a lot more money in world domination.

SCHWARTZ:

[laughter] That’s a very good line.

[laughter]

 

WINARSKY:

Is that on the record?

UNKNOWN:

I saw that movie.

UNKNOWN:

I saw that movie. There’s a bunch of movies like that.

GOLDHAMMER:

Good. A lot at the beginning, a lot at the end, some arrows in the middle. So we'll reconvene in about five minutes and we're going to do a little moveable feast where we'll share each other’s tables.

[Simultaneous comments]

SCHWARTZ:

We get another pass at this, by the way.

UNKNOWN:

Got it.

ZYDA:

Computation.

BOTHEREAU:

This is good. We actually have a lot on here, this is great.

GENIK:

We have to have an analysis and computation.

LYBRAND:

The nice part actually is if you go look at the others, they kind of have the middle.

[chuckles]

 

TWONEY:

So, you know, when you merge the three.

LYBRAND:

There’s boundaries.

ZYDA:

Yeah, we have computation.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BOTHEREAU:

Where are we now?

UNKNOWN:

I'm trying to figure out where we can draw more of these lines.

ZYDA:

We're on brownie break.

TWOHEY:

Oh, we're on a brownie break, okay?

ZYDA:

I think.

WONG:

This gets at kind of that multiple systems idea.

GENIK:

Well, you really would like the ability for any part of the system to talk to any other part of the system if it needs --

WONG:

It should be totally --

GENIK:

It’s like an abstract object class with a generic connection.

TWOHEY:

It’s called a phone. You pick it up, you dial it.

GENIK:

Generic computation.

ZYDA:

It’s an API –

[Simultaneous comments]

TWOHEY:

Phone numbers, yeah, they exist. I'm not being funny. Like seriously –

BOTHEREAU:

You don't need a Web portal to – yeah.

TWOHEY:

You need to talk to me. Like you have my phone number if you'd like to call me up. If I can answer, I'll talk to you, right.

WONG:

I guess I still don't see how – I can see the system being useful for gathering data, I still don't see it being as useful for the analysis part.

ZYDA:

Unbelievably analytic.

BOTHEREAU:

Mike, anything you want to add before we break.

LYDA:

I'd put computation occurs.

BOTHEREAU:

Computation occurs, there you go. Kind of a reminder.

LYDA:

Monkeys and typewriters.

BOTHEREAU:

Yeah. MAT.

THOMAS:

Are these stuck? Do they need to be stuck for the purpose of presentation?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BOTHEREAU:

Yes. But they're not because we're moving them around.

ZYDA:

Don't move the page.

Team Activity: Identifying the Human and Technical Requirements

UNKNOWN:

Plus you analyze what’s going on.

UNKNOWN:

That already exists, right.

GENIK:

And it probably has some inputs of its own that …

SCHWARTZ:

What’s different with that than Jason?

UNKNOWN:

Younger people.

SCHWARTZ:

Younger people.

UNKNOWN:

It’s not the same people all the time.

UNKNOWN:

So there’s turnover, they learn from --

UNKNOWN:

People, technology and partnerships.

SCHWARTZ:

Actually, at the moment we're doing additions to it.

UNKNOWN:

Additions, okay, great.

UNKNOWN:

Tweaks, little tweaks.

[Simultaneous comments]

TWOHEY:

All of the best people I know in science would never deal with classified stuff because it’s too toxic to their careers and their lives. So you have to be very careful about how you approach this because there’s – Look at this room, right, how many women do we have on the panel? Two?

UNKNOWN:

Actually, there are two on the committee.

TWOHEY:

Two, right. So I mean like already our whole perspective is – it’s primarily like white dudes. This is 2009.

WINARSKY:

True.

TWOHEY:

So we need to make sure that like we have – there’s actually a mechanism in place for projecting like diversity of opinion. It doesn't have to be a consensus, right, like that’s the whole point. In fact, in the future people are going to disagree pretty violently about what it looks like.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BOTHEREAU:

Was there anything on the other tables that you saw that you think – that made you re-think anything in here or you'd like to add?

O’CONNOR:

The middle part.

UNKNOWN:

Yeah.

[Simultaneous comments]

UNKNOWN:

Even though visually there's not a lot, I felt like we had some processing here.

ZYDA:

Yeah, we had computation but automated processing.

UNKNOWN:

Right.

O’CONNOR:

We were much more clear on the inputs and we were much more clear on the outputs and I think the other groups actually probably like hovered in the middle. If it was me, I don't if I'd spend a lot of time talking about it.

GENIK:

Oh, yeah. The only thing I'd add is the parallel. I mean there’s at least one parallel path that is run by the government. We don't necessarily have to tell anybody about it.

SCHWARTZ:

One idea I'd like to add to this is the hypothesis engine from the first guys.

[Simultaneous comments]

BOTHEREAU:

Great. Yeah, feel free to – you want to add that?

[Simultaneous comments]

ZYDA:

That goes on an arrow, it goes on an arrow, right.

SCHWARTZ:

It goes on an arrow and where does it go in our process here?

BOTHEREAU:

From Stewart’s group – yeah.

SCHWARTZ:

Yeah.

ZYDA:

Develop hypothesis for story, is that the engine we're talking about?

UNKNOWN:

Well, there’s these groups.

BOTHEREAU:

Peter, were you thinking of it pretty early on or --

WINARSKY:

Develop hypothesis --

SCHWARTZ:

I'm thinking of it as part of the –

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

[Simultaneous comments]

UNKNOWN:

I want it sort of like part of this.

UNKNOWN:

Feel free to put it –

SCHWARTZ:

Well, actually, that’s pretty much there already, isn't it?

ZYDA:

Yeah.

UNKNOWN:

Yeah.

SCHWARTZ:

Okay, never mind. There.

UNKNOWN:

That’s a good, yeah. Add a good title for it though.

UNKNOWN:

[..?..] Put “engine” in there.

BOTHEREAU:

Any other tweaks, additions, subtractions, improvements?

GENIK:

Do we want to put parallel structure down?

SCHWARTZ:

Yeah, I think we need to put – I'm just going to put it right here, parallel structure.

UNKNOWN:

They're all writing Disruptipedia right now. Putting [..?..]

[chuckles]

 

UNKNOWN:

We've created a monster.

UNKNOWN:

Now we should put it out on eBay.

SCHWARTZ:

And how do we want to define that parallel structure?

WINARSKY:

The parallel structure is the intelligence community side that isn't the open crowd sourced side.

[Simultaneous comments]

GENIK:

It observes everything that's going on in here and aggregates it with everything else.

SCHWARTZ:

And it has other pieces that may not be --

GENIK:

Right.

WINARSKY:

Right.

TWOHEY:

Actually, I think there's another case that's not classified but it is not publicly available either. So you look at – There recently was this attack on

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

a SSL-based Website. I'm not sure if you guys are aware of it yet. So there's a potential to have a little bit of mayhem caused by this thing called client site certificates. And so for certain people, especially on the high side that rely on this, like this is a little bit disconcerting. So you – so the IETF created this like group of people that were talking about how they're going to handle it or a DNS vulnerability last year, right. This wasn’t – didn't need to be classified but you wanted to have some things that you wanted to talk about that were not necessarily distributed.

WINARSKY:

That's interesting.

TWOHEY:

The minute something's classified.

ZYDA:

Confidential –

[Simultaneous comments]

TWOHEY:

You have – there's a whole bunch of like federal and legal baggage that comes along with it and like maybe – you want to have two parallel ones. You want to have a classified one and you want to have one for some things that are must maybe not for the general public.

[Simultaneous comments]

UNKNOWN:

Private.

[Simultaneous comments]

SCHWARTZ:

Why don't you put on here "classified" and "private."

WINARSKY:

And private groups, private discussions.

WONG:

Do we want to put anything into the technology around something like a – sort of an intelligent Web spider or something –

[Simultaneous comments]

WINARSKY:

That's over here.

WONG:

Do we have that?

WINARSKY:

Web crawler.

UNKNOWN:

We have it on input.

WONG:

But as a technology arrow.

SCHWARTZ:

So now we're ready to move on to technologies, people and partnerships.

BOTHEREAU:

So that’s good, yeah. Now we can shift, people, specific technologies, and partnerships – organizations, institutes, entities.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

WINARSKY:

Some people went in to the technology –

[Simultaneous comments]

UNKNOWN:

A post-it note maybe.

[Simultaneous comments]

O’CONNOR:

We could go to each note or each orange thing on the table and write how many people and who for just each one of them. If we want to enumerate the options or --

BOTHEREAU:

Yeah. So we have a couple pads of post-its. So for people that could mean either teams of analysts inside the government, how many it should be, how they should be convened and structured but it could also mean specific individuals or groups that you might be thinking of already that do this.

GENIK:

Is there something that the NRC could do that it's not doing already? I mean with TIGER and other committees.

SCHWARTZ:

Yeah, a question for me is exactly that. What distinguishes – For example, I define this Committee on Disruptive Technologies. What is different about this from Jason, from TIGER, from other entities that have been charged with similar tasks, not identical tasks?

GENIK:

Other than they have a different charge.

SCHWARTZ:

Yeah.

GENIK:

They're a part of this group.

WINARSKY:

So these people – this takes a group of people over here to develop hypotheses, that's what they did over there in that table too. These people differentiate –

[end recording]

 

[following are very rough notes on topics mentioned after recorder shut down]

SCHWARTZ

mentioned high levels of knowledge. Suggests a partner might be the Smithsonian. They would like a new source of revenue.

LYBRAND

suggested a kiosk.

WINARSKY

suggests a museum of disruptive future threats.

O'CONNOR

mentions the insurance industry.

TWOHEY

talks about the X Prize and Netflix' automated processing model.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SCHWARTZ

emphasis regular outputs, mini movies.

WINARSKY

suggests partners might be movie producers.

TWOHEY

asks is celebrities might fit in.

LYBRAND

suggests getting data from downloads.

WINARSKY

suggests TED as a partner, others contribute TED support and examples.

LYBRAND

says Disruptipedia will take the longest.

TWOHEY

estimates twenty groups for $50,000.

BOTHEREAU

asks what can make people feel special on Disruptipedia.

TWOHEY

cautions against YouTube quality movies.

SCHWARTZ

reminds that there's always the occasional genius.

Feedback to workshop participants in Appendix D.

GROUP 2

Group 2 Participants:

Moderator:

Carolyn Mansfield

Steve Drew

Jennie Hwang

Gilman Louie

Philip Koh

Bill Mark

Mark McCormick

Phil Nolan

Ben Reed

NRC Staff Member:

Kamara Brown

Team Activity: Designing a Scanning System

HWANG:

Yeah, I just want to follow up and say, you know, you use one word, “rely”, you know. No, I think this whole thing, process, the key, you don’t rely on any one of feedback. It’s really hard to use as a reference point. Not rely. That’s I think the difference from what I see. You don’t rely on any visionary, you don’t rely on anyone. You really look at integrating or --

GILMAN:

You don’t rely on the…

HWANG:

You don’t rely, so that’s the key word you used. I don’t agree.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

UNKNOWN:

So --Gilman saying there are two different pillars and that may be one of the things that we’re going to have.

McCORMICK:

The only thing that concerns me a little bit about those conversations, I think a lot of the stuff we’re talking about works really well for something that’s going to be sort of passive architecture. It doesn't work particularly well with something that is a small, seemingly insignificant, that creates a massive issue such as -- You know, one that I was thinking about, a couple years ago we published in Popular Science, an EMP ball in the middle of New York City, you know. And let’s face it, you can go to the local hardware store and get all the components to basically make one of those damn things, you know? Pretty damn close. But by the same token, you know, you start thinking about the implications of what that’ll actually have at the end of the day, you know, it’s a huge implication. 3340

GILMAN:

Yeah. I was thinking about what Stan was saying, earlier, this idea of taking stories and have -- One way you could do it using a narrative is you say that, you say any point of entrance to your country and you place say tell me what is that? If the world was to go right for you or rotten in the next 60 years, tell me your best possible vision of that, tell us the most --

McCORMICK:

Take the extremes.

GILMAN:

Take those polar extremes and then use that as a way to say okay, how do we build maps? You get very different extremes.

[Simultaneous comments]

STRONG:

There’s some other ones like that, the outliers.

[Simultaneous comments]

NOLAN:

Is it anybody or is it in fact particular types of people in that country?

GILMAN:

You might treat a government leader different than a random person who walks down the street.

McCORMICK:

It does give you a context because that’s, I mean, that’s my big issue that I see, is like I see a lot of the stuff that we’re doing right now in the States is very, very U.S. centric and the reality is that the, you know, the realities -- I mean, I spent most of my life living overseas and the reality is the rest of the world, I mean, like I was born in Zimbabwe. Talk about a complete state shift. I mean, Mugabe came into power and it’s a completely different country, you know.

GILMAN:

So they could be anybody. The question is can you segment it and then do all the demographics and map it out exactly.

McCORMICK:

Yes.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GILMAN:

For example, asking somebody who lives in the Midwest, you know, just a family member, it might be a very useful exercise, maybe asking a slightly different way. “What’s the thing that worries you most and what’s the thing that you really would like as your outcome?” Understanding that cultural base and helping with bias mitigation, say, “Hey look, what’s going to be technology bias or poly bias or Western bias, you know, asking” –

[Simultaneous comments]

McCORMICK:

You know, one that would actually help with that is wisdom.

LOUIE:

Wisdom. Ohh.

McCORMICK:

Yeah, there’s a guy -- linguistic mapping. I guess you’d pull something up on that one. I use it a lot in what I do. It’s called neural linguistics and there’s a specific thing around it. But there’s a guy by the name of Dr. Michael Hall that came out with one. Think about it as like Meyer’s Briggs on steroids. There’s 64 different maps and what’s really cool about it is it maps out how you think, how you process information, but more importantly, are you the kind of person in the kind of society that makes decisions first and then thinks or are you the kind of society where you go through and -- you start to really get very predictive on how people operate on a daily basis, you know. I mean, it’s just one of the ones -- And the nice thing about it is you can use a lot of software tools these days for looking for different types of key words to figure out all right, there’s an 80% probability rate for this and things like that.

LOUIE:

Interesting

McCORMICK:

Pardon me?

LOUIE:

[unknown]

McCORMICK:

No, actually this is pretty universal.

LOUIE:

Across all the languages?

McCORMICK:

Yeah. I mean, but the behavioral characteristics that go underneath it are –

LOUIE:

Are the same but the mapping.

McCORMICK:

Are the same. Now obviously the tools, which you’re using for some of the stuff, but the characteristics are universal across all languages. The tool you’d use and how it gets implemented obviously depends upon the actual language patterns themselves.

STRONG:

There’s another set of drivers that we find are often really useful when you’re trying for breadth and that is take the accepted trends and assume the trend stops. And I’ll give you a really good example, is there’s a general demographic trend that applies almost uniformly across the globe and that is urbanization. Urban areas turned out more dense, populated, and nonurban

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

areas are getting less densely populated. That’s the trend, that’s the accepted trend. There’s no particular sign of that changing. What would happen if that changed? And then we again back test because –

LOUIE:

That’s an interesting approach. You could see that it’s different.

[Simultaneous comments]

McCORMICK:

I’d almost say polarized, like you can actually take it, take the trend to the irrational extreme and then take it to the other end.

STRONG:

Yes, yes, yes, exactly.

REED:

You know, that’s exactly what science fiction [..?..].

[Simultaneous comments]

STRONG:

Yeah, yeah, yeah, that’s what they do, yes.

[Simultaneous comments]

REED:

I can tell you. Why don’t we just alter the corpus of science fiction, analyze it and then figure out which ones are going to come across. Then you have your narrative. So then we need to write up your report and you just summarized your book.

LOUIE:

That goes back to my “Star Trek” comment.

REED:

Yeah, no, I mean, really.

NOLAN:

‘Cause it does feel like there’s a lot of ideas we have around idea generation. And --

REED:

And if everybody…

[Simultaneous comments]

NOLAN:

Well no one want the -- they’re one of the funniest parts of this. When it comes to the evaluative piece which in my mind is always kind of the hole in the middle because you say technology and smarter people and they’re kind of vague about that. I’ve heard them. Mark, are there any other?

[Simultaneous comments] Yes.

STRONG:

We do have very specific techniques that don’t go to consensus. And the reason --

McCORMICK

Intentionally

STRONG:

-- intentionally but partly because it’s expensive and we need techniques that work without going to the expense of reaching consensus. So there’s a

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

technique that we use to evaluate, okay, and we’re almost always evaluating relevance. So we’re doing a relative evaluation usually on a five-point scale. And in particular we’re usually evaluating impact to the client and uncertainty, what we know, how much we know about. And both of those on a five-point scale, relative five-point scales. So when we ask people to do that evaluation, we do go to experts. We also go to generalists and my experience is that if you only use experts you do not get good results in the evaluations.

LOUIE:

CIA did a study and they call it, generalists are called the journeymen, but the journeymen have -- An apprentice doesn't know enough and experts always too much good. Journeymen ask the right questions but also are smart enough to pursue it.

STRONG:

Exactly. Okay. But, all right, so we use a technique that I learned on programming committees back in the sixties and seventies. The technique is you ask a question, you know, for a, you ask for a rating, okay, and so you ask your group, small group, and every time you ask a question you also ask each of the people to rate on the same five-point scale their expertise to answer that question, self-reported expertise, okay? And you use that as a weighting for weighted voting and just take the weighted vote as the result. And it’s not consensus, it’s not the same as the real spent the time to go to aggressive voting. But it’s – [General laughter] – it seems to get -- Look, one of the things, if we can –

HWANG:

It’s democracy.

STRONG:

You know about the definition of a consultant as the one who sees himself as the smartest person in the world? That’s the --

McCORMICK:

He’s not going to be a consultant very long.

STRONG:

No, no, that is -- Not that he says so but from his point of view or her point of view --

McCORMICK:

I just know when it’s a consulting firm, you’re not going to be in business very long.

STRONG:

Okay, I --

LOUIE:

There’s another technique which is very a variant backcast, which is, you say okay, here’s an alternative future, okay? So you ask a group of experts, you know, draw us the roadmap of the future. So there are a couple versions of the roadmap. Roadmap number one assumes there are no miracles, assumes no miracles, right? So using current trends – Moore’s law

[Simultaneous comments]

LOUIE:

-- extrapolating – but how do you get there, kind of given what’s known? Well now if there’s a miracle, how could this disrupt the pathway to get to that alternative future? In other words, would it speed it up? What are the

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

things that could stop you from wanting to get there? Yeah, right. And the third one is what’s the show stopper? Something that comes along that says well, that’ll never happen because there’s something even better, an alternative, that, or it’s an impossibility that can’t solve, basically can’t solve this problem, can’t get there. It’s a variant to, a backcast, that says do three maps for every backcast, the natural, most likely way it’s going to get there, one that has, you know, maybe one and a half miracles or a couple minor miracles, and what’s the roadblock. And then be able then to use that as the foundation in the track and do an evaluation cause if you can’t get there, then the evaluation won’t work.

STRONG:

But you create a roadmap with lots of roads and lots of points on the map.

NOLAN:

So let me ask the tough question, which it’s my job. There’s a lot of ways of idea creation I’ve heard which feel like they can be automated, can throw out tons and tons of possible ideas. The evaluative mechanisms I’ve heard feel very labor intensive. What you just described felt like a good system. It also felt like one that can’t scale as fast as we’d be able to scale to create these ideas, either as –

[Simultaneous comments]

STRONG:

Oh, the system I was talking about is totally automated.

NOLAN:

Totally automated?

STRONG:

Yeah.

NOLAN:

Individuals have to –

STRONG:

Yeah.

NOLAN:

-- evaluate

STRONG:

You send out email and you say here’s the question, here’s your -- You send, you return, you respond with your weightings and that’s it.

NOLAN:

But that is automated?

LOUIE:

Yeah.

NOLAN:

For example, what Ben was talking about as sifting through the corpus of science fiction to spit out ideas, it’s going to be creating ideas faster than –

STRONG:

That’s also automated.

NOLAN:

I know but that --

REED:

No, the generation’s automated but the evaluation, you’ve got a lot of people.

STRONG:

The evaluation involves people.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

HWANG:

That’s more simple than automation.

[Simultaneous comments]

NOLAN:

I’m not saying manual versus automated. I’m talking about speed of creation. I hear the idea -- when we think of idea creation, we come up with some high speed ways of doing it.

STRONG:

Harvesting will certainly do more.

[Simultaneous comments]

NOLAN:

And the evaluative feel like they’re even slower and I wonder if there’s something we can do to speed it through.

REED:

Well I think if you structure the prediction correctly, so that if you had some trigger conditions then you could do things like mine the news, right, or look at current reports and just find anything, basically, right? Get all the data you can from that area that you’re looking for and just correlate it with the predictions. And then that will give you -- ‘Cause predictions that don’t have their conditions satisfied you don’t need to look at. It’s only the ones that you have --

NOLAN:

the conditions -- No, I think it’s the temporal problem, which is if you’re mining the news, you’re talking about what’s known today and –

[Simultaneous comments]

REED:

No, no, but you can get trends.

STRONG:

If you’re talking about signposts, that’s what you’re actually [..?..] –

[Simultaneous comments]

REED:

No, no, like for example --

STRONG:

No, he’s talking signposts 'cause he’s talking about things that it’s an event that either happens or doesn't happen it in advance. Signals –

LOUIE:

Signposts beforehand to say what’s the signal that I’m going to need to match against the signpost? The problem is not the signals – well there’s a lot of stuff in there. It’s not signal and maybe not even the vision, but how do you --

STRONG:

Signals include those anomalies.

LOUIE:

-- how do you create this, so how do you create those roadmaps in a way that’s not labor intensive, how do you create those signposts to say these are the things you should be tracking?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

McCORMICK:

Well see, it even goes beyond that. It’s like I’ll give you a specific example. It’s a project I did for Wal-Mart and we were doing a pretty big analysis of the entire stores and what’s going on and stuff like that. And, you know, they’re already really good at selling individual products. What now matters is what’s the combinations, right? I think this is where it comes into other things. So I’ll give you an example of an anomaly that, one of the things that came up that just shocked us – and there’s a couple hundred of these that came up. The correlation between the purchasing of, you know, friendship cards, so somebody goes into, and gets a card of “I like you” kind of thing and the sale of condoms was like through the roof, you know. Sorry. I’m using an extreme example of a situation. But then there’s like how do you --[General laughter] But it wasn't obvious to them and so then it’s like you take those type of cards and put them over by condoms, you get lots and lots of increased sales. You put it the other way around, it doesn't work, you know. I’m using that as a simple example but to a certain degree, you’re kind of implying this is well from a macro economic perspective, a macro situation. You’re saying okay, where are these relationships and those actually tend to be the trigger points that have far more impact on society than the other ones, like okay, let’s track green cards and let’s track condoms separately, you know.

NOLAN:

But some people at this table we haven't heard from much in this discussion, and I want to make sure you guys have a chance, so…

[Simultaneous comments]

[Inaudible comments]

DREW:

This strikes a resonance with me. You may be looking for, or even looking at the innovators come up with, guarantees your X. ‘Cause they became innovators because of some segment criteria, which allowed them to lapse. What we haven't really addressed to any great extent are those criteria and those things that allow innovators to rise and innovate. For every innovator that rises there must be ten or a thousand --

NOLAN:

That don’t.

DREW:

-- that don’t. But in order to predict you need to understand I think what the root things are that cause this. For example, an innovator in Bagdad at some point in time said the conditions I’m faced with are no money, no ability to move around, you know, I want big things and I need a solution. And out of that some innovator came up with an IED that does what it does. Another innovator said I don’t have any money. I have plenty of ability to move around. I want to make a humongous big impact but I don’t have any explosives because he discovered pilots who could be recruited and that led to, you know, that was a good innovation. There must have been a thousand lead up innovations that didn't make it in the big time. So I’m wondering if maybe we shouldn't spend some energy looking at what criteria are the lead-to that would allow innovators to be successful.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LOUIE:

With your scenarios you have outlined interesting conditions that caused an action or opportunities about science. Science on the problem spaces and opportunity spaces that people are naturally incentivized to do.

VONOG:

Different geography

DREW:

Oh, yes.

VONOG:

You know, countries like India or Indonesia, they have poor people …

McCORMICK:

It’s an affect on issue is, you know, take anything to an irrational extreme it’s bad but what’s the threshold at which action starts to happen, you know, what’s the point at which it’s no longer acceptable and you hit that tipping point and people want to take action. The only thing I struggle with that is that’s a hard thing to both ascertain 'cause it’s different culturally, as well as do anything innovative. How do you measure it?

NOLAN:

Bill, Carolyn?

MANSFIELD:

Yeah, I was just going to say in response to what you were just saying, I wonder if it’s, part of the issue is that we aren’t able to monitor the communities where there’s a lot of constraints. It’s like my anthropology background but, you know, it’s hard to get into those places, the surveys aren’t effective because they’re not all the time, they’re not reaching people. And one of the things that I started looking at in my old job in recruiting was how to reach people that are, that don’t have access to computers and how do you get jobs to them. And a lot of what we were talking about is cell phones and kind of like instant pulse data, you know, blasting out to people and getting information back. And I wonder if there are just other ways -- you mentioned before that surveying wasn’t working and, you know, you can’t get to these people. I wonder if mobile technology and other things that haven't been used for surveys or data collection before are ways to start accessing those populations in a way that you’re surveying but you’re getting mass numbers of people to get data really quickly.

MARK:

Well so again I’m, I was struck by the IED example and the 911 example. So to me the, you were making it sound, maybe unintentionally, like there was one innovator who did that. Of course not. It’s communities of people. So for the IED things I still think that kind of thing, what you should be looking for is the experimentation that’s going on. And I think, it’s not, I get the point about thresholds but I don’t think it’s quite that. I think people start seeing what’s successful in solving whatever problem they’re going to work on and then they innovate from there and make it better and better. So the – and I do think it’s possible to get to people who are not Internet connected. I think we should be spending time on doing exactly what you’re talking about, which is thinking about how to understand what those groups are doing, pick groups of interest and try to understand what they’re doing. Final comment, you’re trying to get us to speed up the evaluation part of the story. I agree that that’s a whole. I’m still not getting a good impression of evaluating with respect to what. So there have been some good ideas about idea generation, throwing things out there and getting reaction to them, okay? I like the comment about

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

a roadmap with a lot of roads on it. I don't know what to do with a roadmap with a lot of roads on it. I need to know which roads are the ones that are interesting, the ones I should care about. So that’s, I assume that’s what I’m trying to evaluate, which ones of these things should I care about, right?

LOUIE:

Maybe and maybe not. But there’ll be, pushback. If you have a lot of roadmaps, a lot of roads are bad, but you know where all the stop signs are or all the points are on those good and bad roads. You don’t need to evaluate the roads [loud background noise]… You only need signals as they become true.

MARK:

Right. But you put in a big “if” condition there, which is if I have the signposts that I think are really accurately going to tell me whether that roadmap, whether somebody’s going down that road, I think that’s a very hard problem.

NOLAN:

Well something you said made me think that I’d throw out a very different idea. What if it’s not individual innovators at all? What if the individual innovators put the road in only if there is a community? And so you, one – a need for sure but actually there’s a community, a community of people either supporting them or all trying at the same time.

MARK:

No, they’re just there.

NOLAN:

And the thing is communities have much bigger signatures than individuals.

MARK:

Exactly.

NOLAN:

So I just wondered, I mean, there’s a big “if” proposition there, but --

MARK:

But I think that that’s, I don't know if it’s always true but it’s certainly going to be generically true. I was reading – I’m trying to remember which one it was. I think it was a DSB study that came out recently on this issue of capability surprise and they were giving the example of the birth of flight, of powered flight, okay? And I was thinking what a terrible example that was because there were a community of people working very hard on that, okay? Powered flight was not a surprise to them at all, okay? So the, if other people -- There’s that famous thing in the New York Times about saying that, you know, the hundreds of years before there was powered flight and it was like the same day or two days before --

NOLAN:

Oh, I didn't know that.

MARK:

Yeah, a great story. But, so all that that shows is that that guy didn't know about it. So if we could connect to those communities, which do have a bigger signature than individuals, again, I think IEDs are going to be a great example of that.

NOLAN:

Experts in that area.

MARK:

Exactly.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LOUIE:

Look for the natural communities. That’s what’s interesting about communities. Communities could be groups of people, could be organizations.

NOLAN:

Could be online.

LOUIE:

One of the things we always say is if we ever want to figure out what a company’s really bad at, look at what their slogan is. You know, say quality is job one! No!

[Simultaneous comments]

UNKNOWN:

So what we’ve got for the country is security and jobs.

LOUIE:

Right, your local jobs. Well now for countries, you know, we get, you listen to the political rhetoric, you listen to the speech. In the blogs you look for certain themes where people are constantly questioning the rallying cries of kind of like somebody help me find the answer to this problem. It’s another kind of method of evaluating linguistics and speech to determine what could be interesting problems that people are really going to put resources behind.

NOLAN:

Cap.

LOUIE:

And cap the resources. So it could be anything.

McCORMICK:

I mean, the biggest thing I look for in a company is culture, what is their culture, 'cause that’s more indicative to the success of the company than anything else.

NOLAN:

Tell me more about – what do you mean by culture?

DREW:

What cultures are successful?

McCORMICK:

Well you know, it’s not so much what’s successful, okay, it’s how do they treat each other, what’s important to them, how do they speak, right?

DREW:

Let me give you an example of why I asked? I’ve been listening now all day Bill talk about experimentation. I think it’s the root issue today. You said the same thing when you talk about communities. It’s two different words for the same concept. It’s a company’s willingness to look at experimentation.

CULPEPPER:

Yeah, that’s a big part of it. I mean, like I said, if -- I’ll give you a really good example of a very well known company. They’ve just done a massive culture shift to the negative in the last ten years. And, you know, they were wildly successful and now all of a sudden it went from it was okay to make a mistake as long as you didn't make the same mistake twice, to now it’s not okay to make a mistake. So the level of risk has gone down and the only people that are getting promoted are people who don’t take a whole lot of risk. So the long term viability of this company isn’t particularly good because I know the quality of the managers and executives there and it’s

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

horrible. But yet they have this view that they’re phenomenal. So the subtleties of the culture in these situations.

DREW:

So one of our tasks will be to reduce this to a plan or method. I’d still like to see how you do that. I mean, do you just look at all experimentation in a particular community, okay, all experimentation in a particular community, however finely you define it, and then what do you measure? Do you measure the second role? If there’s a change in experimentation, isn’t that almost more predictive of opportunity than just looking at the experiments themselves?

McCORMICK:

Steve, along those lines I’d actually like to suggest something because one of the things that also kind of concerned people in the conversation is we’re talking about three different areas here. We’re talking about what we don’t know we don’t know, about what we know we don’t know and what we know we know, right?

LOUIE:

We think we know but we’re going to. [Laughter]

McCORMICK:

Exactly. But what strikes me is it’s like it’s almost like the tools for doing each of those is different.

[General agreement]

[Inaudible comments]

McCORMICK:

Yeah, and like really, really early on it’s like how do you just ID you know you don’t know something, you take it in the realm. And there it’s kind of like identify and define and then you get into the whole notion of you know you don’t know something so now it’s, you know, refine and qualify in some way, shape or form. And then you get into the whole notion of what you know you know and that’s all about quantifying, right, and insuring that the fundamental principles haven't changed.

LOUIE:

Or you think you know but you really don’t know which is really simple.

McCORMICK:

Exactly. Exactly. And you’re constantly assuming the assumptions.

LOUIE:

Yeah, for that to happen. And mapping’s really important because you want to be able to touch each one of those lives and whatever your impulse is, so whether you’re following experimentation, you’re following science fiction writing and possibly literature or you’re following, you know, funded experiments, whatever it is. It shouldn't be addressing all this.

[Simultaneous comments]

McCORMICK:

‘Cause I think -- pardon me?

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

McCORMICK:

Like one of the things you brought up earlier like the communities, right, I think the communities fit in the second bucket of what you know you don’t know and you’re trying to qualify and define, right? Whereas more traditional survey techniques are far more better for what you know you know 'cause it’s about quantifying at that point. I think the bigger issue from an experimentation perspective is back in the what you don’t know you don’t know and that’s kind of the science fiction slash experimentation of hey, here’s a theory we have. Let’s go try it out. Let’s see if it’s viable.

NOLAN:

‘Cause see, pushing on the “you”, because -- I thought the flight example was a fascinating one, which is the system at the time, you know, in 1903, if we thought of the system as scientists and engineers around the world, many of them were on the know, many of them hadn't a clue, if we think of the problem at large, nobody had a clue. And I wonder whether the, you know, whether the “you” part, whether that functions when we’re actually talking about a system -- Whatever our system is there’s going to be lots of people involved with it with different levels of understanding and therefore I’m kind of struggling in with how could we even say whether the system was something.

MARK:

Well but the point -- that gets back to this emphasis on openness, right? Because somebody knows what you don’t know. [General laughter] So the point is --

NOLAN:

Yeah, it doesn't exist at all, that.

MARK:

-- you’ve got to get -- The point is you’ve got to get in touch with that. So that could be how do you do the, how do you focus on the experimentation? Back to the what do you do in the 1.0 system, I think you have to pick some communities of interest, some people that you care about, okay, and then focus --

UNKNOWN:

Why would you pick those, on what basis?

MARK:

Let’s not worry about that now. Let’s not worry about that. This is a 1.0 system. I’m going to tell you there’s Afghan tribesmen and Stanford students. Those are the two communities.

McCORMICK:

Actually, can I add one quick thing to it?

MARK:

Sure.

McCORMICK:

I think what you’ve got to do is get rid of political biases that exist with a lot of this stuff because there have been a number of times when, you know, at least in science there’s numerous instances where somebody came up with a great idea and it was squelched for years upon years because –

[Simultaneous comments]

HWANG:

I have an empirical question, you know, need the collective wisdom. You mentioned about to the, how we going to really evaluate, you know,

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

whatever the methodology, what are you going to choose? How to speed it up by using evidence. Remind me, Mark, you were talking about Wal-Mart, remind me about the PIG, okay? They had of course, they had a lot of innovations. Some of them not glamorous, some of them, you know, but they are very much an innovator. They, the innovation come out, per one source, is come out, they really going into the market, now the customers you mentioned, and to see, not just to listen to them, really to read between the lines and observe even the silent language, you know, the body language of those in order to understand what is coming, you know, in the future. So what we are talking about on the one extreme, you know, just to put all the science fictions together and then to see why they come up, the other, of course, to, you know, all the hard data, people feeds and all those kind of things together. So regardless which extreme on the spectrum we are looking at, who are the ones going to evaluate it and how they going to be evaluated, and what are the criteria to be evaluated in order to give some meaningful things to come out. So we are not talking about that at all. That’s kind of the reality to me, is how they’re going to be, you know, you can have all these kind of things together. How are you going to -- that become very key. You know, collecting data is not that, I mean, it’s not a real brainer. [Chuckles]

LOUIE:

You being the system operator, you the system operator, doesn't have to be evaluated. If you can create enough high quality information around it, sourcing of information, it’s up to the user of that information to trust or not to trust the idea.

HWANG:

Well no, just to –

LOUIE:

a possibility which is saying open systems –

[Simultaneous comments]

NOLAN:

I’m getting the word from Daniel we need to stop talking and start writing.

[Mic noise]

 

NOLAN:

Start generation, the evaluation area and try to just put down, you know, in writing on some of the these wacky shapes a couple of these things and then, you know, spend, get that down 'cause there were some really good conversations, and then we’ll start to play around with it.

[Inaudible comments]

NOLAN:

Okay, we’re going to, we’re going to be crazy. We’re going to write. This is –

BROWN:

And just from a distance because I’m not a professional in doing this. One person at a time so I can capture it, please. Thank you.

[Further conversation about recording]

[Inaudible comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

MARK:

I feel that if we don’t start putting pieces of paper down here soon –

BROWN:

And I would be careful.

DREW:

I like this so pick your community. Only today Gilman said it has to have some sort of human and Dan was talking about science fiction, and help drive a successful science fiction writer to find his or her abilities to find human relevance.

UNKNOWN:

Right.

[Simultaneous comments]

DREW:

[..?..] this is tied to human, the human condition or human relevance.

[Simultaneous comments]

DREW:

Communities are human relevance. That is what they are. They’re doing something.

UNKNOWN:

We exist because of [..?..].

MARK:

They exist for lots of reasons but here they’re trying to solve some problem, right, that’s what we’re interested in.

DREW:

Here’s why.

DREW:

So I don't know how you –

BROWN:

Okay. Well they’re going to take pictures I think for that. So I need to kind of capture for the transcriber what you guys are saying so she can put it together, this – it’s not going to -- so I think that’s not…

DREW:

Actually this feeds into say pick communities, it feeds into harvest.

BROWN:

It actually does, yes, it does. Social networks, blogs. It’s the same thing.

MARK:

So the picked communities –

[Simultaneous comments]

DREW:

So this is the over level.

NOLAN:

Yeah, yeah. These are high level concepts, brainstorm and what does that say?

MARK:

Idea generation by a contest worldwide online, weekly winners. Audience is Hollywood and Bollywood.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

NOLAN:

So this is games for idea generation, right? And the high level word there is “games”. Just give it a title.

BROWN:

Now are these two the same, part of the -- No, no, harvest, harvest and this part of the same system?

DREW:

Let’s discuss putting them in an order we agree on.

UNKNOWN:

Yeah.

DREW:

Harvest is from these, right?

MARK:

Right. Well harvest could be from both, right?

BROWN:

Yes. That’s right.

MARK:

We haven't said what we’re doing with communities so we have to be observing them or doing something with them, watching them experiment --

MARK:

Well we could do ethnographic studies of communities.

DREW:

So don’t move it. Get it -- don’t, don’t put your pen away. Say what you just said. Watch them experiment. I don't know how you want to say it. But I mean, you pick your communities, you generate community surveys. But the key issue is you watch them experiment 'cause that’s, from the experimentation will arise the innovations. Self-validating, right, or self-

MANSFIELD:

Sorry. I think that falls under generation.

STRONG:

Mobile survey data collection. Okay. So this is the survey idea.

DREW:

Maybe it’s part of that, huh? As far as those are the same kind of things

STRONG:

I would stick it in there.

[Simultaneous comments]

STRONG:

Harvest, yeah, this is harvest, that’s harvest

[Inaudible comments]

[Laughter]

 

UNKNOWN:

Reapportion

UNKNOWN:

No, no.

[Laughter]

 

DREW:

Well we’ve got to be able to read it.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

MARK:

We were doing, yeah, we were doing the high levels up here. Spelled it out said okay, goes out there somewhere.

NOLAN:

I’m throwing out the third category as a possibility which is we generate ideas, which is, it’s scanning in a whole bunch of other things, we’re evaluating them, make some sort of prioritization. There’s a communication piece. I don't know getting to a customer or involving people, but I feel like there’s a handful of ideas we’ve had around some sort of communication piece.

LOUIE:

Is that valid?

NOLAN:

Well there’s something going on. [Chuckles]

MANFIELD:

There’s definitely -- yes.

MANSFIELD:

There’s a narrative.

NOLAN:

Yeah. I’m going to put down here some of the things that we’re…

[Simultaneous comments]

STRONG:

But it’s, look at some place we’re going to do the scenarios, we’re going to --Oh, let’s see. Okay. I will just put out what we do, okay, so we do trains and but let’s call it transportation

MANSFIELD:

You feel like there are added levels of complexity you want to put in there? Which shapes?

STRONG:

no, we’re just stacking them.

MANSFIELD:

Good. Yeah, exactly. And then we get to get the string involved?

STRONG:

And then we have something called. So these go here and the signposts, this is recognizable.

McCORMICK:

What? For what? What’s defining the cast?

McCORMICK:

So it’s what’s technology and what’s…

[Simultaneous comments…]

STRONG:

Yes, yes, indicators, yes, and these have other names too. We have to get the narratives, you know.

[Simultaneous comments]

NOLAN:

I tell you what, since we have a lot of ideas down here, one of which, all of which are stunningly good, why don’t I ask different people to just spend a few minutes, like spend 30 seconds, read yours and say kind of what it is.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

And that might help us figure out where to categorize it. So who did this thing top left?

MARK:

Pick communities? Pick communities.

NOLAN:

Who wrote that?

MARK:

I did.

NOLAN:

Got it. Who are you? What’s your name?

BROWN:

Bill.

NOLAN:

Bill, cool.

BROWN:

Are you Bill?

NOLAN:

Okay, what’s that mean?

BROWN:

Speak up, please. I can’t hear. Be a cheerleader.

[General laughter]

 

MARK:

No.

BROWN:

Okay, well don’t.

MARK:

So I think that what we should be doing is choosing to focus on a few communities in Version 1.0 and look at what they do as a source and to look at that as a source of what’s going to be done.

NOLAN:

Cool. Another one. Who’s this?

BROWN:

Don’t be shy.

DREW:

This one goes across this board from here to here and it simply says “iteration.”

BROWN:

Who are …

DREW

Steve.

BROWN:

Yeah, I got you. I remember.

DREW:

So this is iteration but it’s something that you have to tie together and it says to the harvested visions and experiments so you’re harvesting here, you’re data mining here, brainstorming here and the experiments to realize these visions meet the communities, that’s these guys up here. If the answer is yes you continue on to I think this one, you continue on to this. If the answer is no, then there’s something wrong with this earlier process. So I’d put that there and I’d leave yes to that and no to changing the system.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

NOLAN:

Okay, good. I read this one up top, Bill, games, this idea generation via contests worldwide, online, weekly winners, audience is Hollywood, Bollywood, Sand Hill Road and the IC, as a way of generating ideas. Pretty straightforward.

MARK:

From other sources.

NOLAN:

From other sources. Another one. “Multi”. Who wrote that?

HWANG:

Yeah, yeah, me. Multi-narratives from same set of data on environment, and they are always subject to interpretation. So if we would take one narrative, probably we’d be limited upon reading it to me, so we should have multiple narratives and that way we’ll see they integrate on every area.

NOLAN:

That actually feels a bit like scenarios that I hear people talking about.

DREW:

That is a lot like scenarios and we’ve got scenarios down at the other end.

NOLAN:

Cause I can imagine it both in a, you know, idea creation area and the communications. Who wrote the one that says “data mine”?

LOUIE:

Yes, this is data mine the Internet for new concepts and terms and go out to all the tag clouds and just start finding new words and terms and mergers.

STRONG:

Oh, anomaly detection is something you can do automatically, assuming you have things to watch. So when do we have things to watch?

NOLAN:

Interesting 'cause it’s both evaluative and idea generation. You can make a new area if you like.

STRONG:

Well anomaly detection probably goes here and down here.

[Simultaneous comments]

MANSFIELD:

Well is anomaly detection under evaluative?

STRONG:

Well it doesn't have to be 'cause anomaly detection could be automatic but it can also -- see, you have to decide on the threshold so, and that’s judgment and the judgment is dynamic. It would change depending on what else.

[Simultaneous comments]

NOLAN:

Signposts? Are those the conditions for the predictions? Is that what the signposts ?

REED:

Yeah, those are necessary outliers that say that this is becoming more true.

[Mic noise]

 

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

STRONG:

A signpost has to have both a recognizable potential future event and an action, a recommended action. If it’s not actionable it’s not a signpost, it’s just a signal(?).

NOLAN:

Oh, yeah, so this is -- I’m talking just conditions.

STRONG:

So these are signals or indicators, yes. Signals, indicators. The signposts are way down there.

[Simultaneous comments]

NOLAN:

Who wrote these? Same handwriting.

STRONG:

I did.

NOLAN:

Excellent.

STRONG:

Brainstorming and harvesting and this is deep dive into harvesting. This is harvest social nets and science fiction with probably dot, dot, dot.

NOLAN:

Who’s brainstorming?

STRONG:

I wrote that.

NOLAN:

No, who is doing it?

STRONG:

Oh, who does the brainstorming? Okay. In general, in general yes, you can go to specific communities, you can go at random, you can go to experts. I like to do all of the above. Because what I use for brainstorming is that ideas have to all be expressed 25 words or less and the more ambiguous they are, the better.

NOLAN:

Can you put some of that down 'cause that’s a great set of information.

STRONG:

Yeah. Okay.

NOLAN:

What’s this one?

LOUIE:

This is an add-on to harvesting, which is specifically take a look at what I would call summaries of popular media, books, TV, movies and games. And you can go to a place like Amazon or anyone that and take a look at the summaries and extract so what are the classes and themes in there. You look and you’re going to say hey, you know, what’s the plot line here, and look specifically for technology as well as human conditions source.

NOLAN:

Who circled that?

REED:

Yeah, so actually this is along the same lines but it’s to correlate the predictions. You data mine this….

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

NOLAN:

So is what you get from this activity, is it a set of raw material ideas or is it actually the evaluation?

REED:

Yeah, this is the evaluation.

STRONG:

That could also be a signal, right, because that correlation creates a signal.

NOLAN:

I wonder whether it applies to the market.

REED:

Well so I kind of viewed the idea generation just as a bunch of words and then your evaluation. You need to come up with a list, right? So at least rank it.

LOUIE:

Track it, kind of track it.

REED:

Yeah, exactly.

NOLAN:

So I think we’ve got that, we’ve got that. There’s a lot of love down here we haven't seen. So everybody can hear, you want to slide down a little bit and we can get… Can’t we do both? Then we can just talk about it. Okay, I wrote this one. Hi. It says plead evidence of a community of interest to make an idea plausible. One evaluated technique to say is there a bunch of people working on it or just one. Okay?

REED:

But do you need more than one?

NOLAN:

Do I need more than one what?

REED:

One guy?

NOLAN:

Well I think my argument was this, just totally me making it up, is that you need one person to invent something but it’s plausible when there are dozens of people trying to work on the electric light that somebody won’t get it. It’s less plausible that [mic noise] without any context that somebody will suddenly come up with it. So I think the signature of invention --

MARK:

Plurality of the idea of what the

[Simultaneous comments]

REED:

Yeah, but the disruption is when the invention’s used, not when the invention’s made, right?

STRONG:

Yes, yes. Correct.

UNKNOWN

[unknown]

NOLAN:

Yeah, yeah, yeah.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

REED:

So what I was arguing is if you have the brilliant idea from science fiction and you say, “Boy, is there anybody actually working on the, you know, morphing the cat brain like in science fiction?” and if the answer is yeah, actually dozens of people around the globe are playing with it, I’ve got a community story is more plausible.

LOUIE:

Either dozens are fiddling with it or it’s attracting some form of resource, other human beings or it could be dolls. It could be some sort of application or look at this algorithm. Could be a missed idea. Stealth, right, and it attracts –

UNKNOWN:

the Geoorbital Air Force satellite.

LOUIE:

That’s right. Or it …

[Simultaneous comments]

NOLAN:

I notice a lot of fine graphic artwork. Who is responsible for this great artwork?

McCORMICK:

Me.

NOLAN:

Off you go, Mark.

McCORMICK:

All right. I’ve got a couple here. A couple of you probably know lifecycle, Gartner, something like that, where like as stuff is coming up here, nice thing about it is you’re going to know about it beforehand before it really becomes like. This is one that people probably don’t know. It’s innovation, well it’s actually a different triangle.

BROWN:

Could you – I couldn't –

McCORMICK:

The theory is that there’s four fundamental ways to differentiate it, innovation, performance, breadth and cost in every single market for this through a natural cycle. And basically --

NOLAN:

Could you give an example so that everybody –

McCORMICK:

So let’s take cars. When cars first came out, it was horse and buggy; they were very innovative. And then basically it was all, you know, key characteristics, how fast does it go, things like that, performance. Then all of a sudden brands became really important, right, so it’s a one-stop shop. I can go to buy a truck, I can go to buy this but I know in this one it’s going to be good. That’s the breadth category. When you get to cost it’s all about that is the most important thing. Now here’s the dangerous thing about it. At each of the four corners, sorry, three corners, it’s a really critically dangerous place to play 'cause if you’re the fastest and somebody comes out with something faster, you’re dead. There’s always sort of the inverse as well because there’s the fastest, there’s the slowest, there’s the most expensive and the least expensive. It’s like Rolls Royce actually plays in this.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LOUIE:

But you want to be an extreme.

McCORMICK:

Yeah, you either have to play at that extreme or not. Now here’s the other place where -- You get lots of disruption going from here to here and from here to here, it’s when the natural market shifts that you get a massive level of disruption from here to here. So think about like the horse and buggy to all of a sudden the car. The car came out and all of a sudden you went from horse and buggy playing down here to suddenly, you know, some radical innovation.

MARK:

What you were explaining to me, an example I thought that was fascinating, Mark, was the disruptions within a triangle are often to the players outside and the disruption when you hear the reset can be to the inside.

McCORMICK:

Correct

LOUIE:

Yeah. That is very good

[Simultaneous comments]

LOUIE:

So can you write that theory in the back of this book so I can capture it.

BROWN:

Does it have a name?

McCORMICK:

It’s actually called a peer team.

UNKNOWN:

Yeah, kind of ride on the backs of this–

McCORMICK:

And I’ve got a bunch of that stuff for us.

[Simultaneous comments]

BROWN:

What was theory? What was the name of the theory? Mark, what was the name of the theory?

[Simultaneous comments]

McCORMICK:

All right, next one really quickly, Ansoff matrix, which is basically markets and technology. So new markets need new technology, etc. What’s interesting is the level of disruption. When you have a new technology going to existing markets you have a certain level of disruption that takes place. When you have a new technology going into new markets you have another order of magnitude, longer adoption cycles, all kinds of things like that. Just one other model to look at! And here’s a classic one, Jeffrey Moore, you know! What is the technology, what exactly is that as is that ? What’s it going to take to get?

LOUIE:

Is that a tech or an application?

McCORMICK:

Pardon me?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LOUIE:

a tech not an application.

McCORMICK:

Yes. Metaphoric analysis, I’ve got a bunch of stuff to show you. This is the analysis, all right, so 64 different categories, you know, you can start to figure out how people think and how they react. And the last one I had was sort of profit motivation and scenario analysis, so what exactly is motivating people to do what they do at the end of the day and getting down to --actually, this goes with this last one. Sorry. Second order analysis. What I find is combining these two is really powerful.

NOLAN:

I understand second order analysis conceptually. Is there a process that automates it or allows it to happen more quickly?

McCORMICK:

Well it’s, honestly, it’s going out and asking somebody, “How do you feel about X? Now how do you feel about Y?”, all right, and you get something really weird -- I mean, head goes to the side and … They had a complete picture.

NOLAN:

Or it may not even be just them. You say so many -- A survey says that X number of people think about this is in a certain way. How do you feel about that?

McCORMICK:

Now how do you feel about feeling about that?

NOLAN:

Yeah, yeah, wow.

McCORMICK:

It’s that second, so it’s not just, you know, I have a feeling about those people but it’s not like a. It’s actually .

NOLAN:

Cool. Thank you. That was very nice. Okay, Gilman, I noticed you’re . Wait, it’s Phillip.

KOH:

Let me just -- I think basically talking about measurement of impact. You know, I think in the sense that you kind of identify, you know, those having the strongest impact or moderate or the least impact. Help us to identify, you know, a real estate can move forward. Market survey, I think that’s, primarily I think that’s one of a real estate so have to kind of, you know, collect ideas, you know, identify way of expectations from the consumers or from the market itself, you know, what are the refinements required. So I think that’s kind of more, I guess it’s more the evaluations of.

NOLAN:

Gilman, you’re a here.

LOUIE:

Oh, this goes way down at the end but --

NOLAN:

Why don’t you tell us about it.

LOUIE:

It’s impact analysis, is understand a day in the life of, a day in the future life, which is basically –

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LOUIE:

It’s playing the scenario out for the crowds.

NOLAN:

Give me the context.

LOUIE:

Yeah, give me the context.

STRONG:

But that really is, that’s really where it goes.

NOLAN:

What about some of these other ones?

STRONG:

Okay, so scenarios impact analysis, thinking, the envisioning. This is human work that needs to be done. This is not machine work, okay? On the machine side we can recognize, once we know from here what we’re looking for, so we’ve got to have, for this kind of thing we have to say, you know, we can’t just look for everything. We have to know what are the search terms that we’re interested in. We can then find technology trends and specifically measure. So the measures of interest are things like the energy density of batteries, and you can measure that several different ways. It’s actually many dimensions have. Okay. So between this and this there’s a whole bunch of techniques for turning the crank generating signposts. You need the measures of interest in general but there are ways to get signposts without measures of interest. The signpost is a recognizable potential future that also has a recommended action. It’s not actionable, it’s just a signal, it’s just an indicator, it’s just some place on a measure of interest. Okay, with signposts you can then synthesize what we could call that roadmap with lots of roads, okay? This is a prioritized vision network with signposts, okay? And visions are the things that correspond to the scenarios. There are scenarios for visions actually. And this is actually a two-dimensional path and the two dimensions are impacting, this is kind of reinforcing that point.

NOLAN:

So let me ask that we do -- Before we organize any further and I have a couple ideas how to do that, let’s --

McCORMICK:

Actually, something that Ray had actually talked about before that was really important was, you know, the extreme analysis.

NOLAN:

Yeah, yeah, yeah.

McCORMICK:

You know, and adding that in with that 'cause I think that’s actually, it actually defines what the game plan.

STRONG:

It’s another of the techniques.

STRONG:

There’s a whole bunch of techniques that.

[Simultaneous comments]

NOLAN:

So what I’m asking everyone to do before we do anymore organization is some of these activities or concepts are those which can only be done by humans, vary, and I’m going to call it manual but done by humans for the H.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

The humans are the real critical factor. Other of these are more automated. Some are going to seem rote. Yes, can we just go through and if you’ve written one of these either write a big H, a big A or if you can’t decide, put both, H and A.

MANSFIELD:

We could also mark it in different colored stickies or something.

[Simultaneous comments]

STRONG:

Semi-automated is really the best we can do for these.

NOLAN:

Human, automated or it’s actually, it has to be one or both. And just try to go for one. You know, it’s an extreme but push it as far as you can. Then we’ll organize.

[General Conversation while doing task]

NOLAN:

Our goal we have here are two things which are very difficult. Number one is make sure we’re oriented only in the looking only in this direction.

MANSFIELD:

So this is going to be stuck on an easel to present. Does that make sense?

[Simultaneous comments]

NOLAN:

Wait, wait, that’s an easy one. The slightly harder one is we want to do more -- Right now we have like a few big clusters. In a couple places I’ve heard people talk about some interesting A to B connections. The A to B connections are the ones we want so we’re going to be rebels and we have glue sticks and somehow make it happen and [noise] Carolyn knows how but I sure don’t.

[Side comments]

 

McCORMICK:

It’s so much of this whole thing of how much of this, which of these break out and how they break out between what we know, we don’t know, what we know we don’t, you know, kind of thing and start to break out. It just strikes me – otherwise the context is, we’re mixing it all together.

[Side comments]

 

NOLAN:

Let’s go through some connections, right? So what kind of [mike noise] --You watch people, you pick communities, you generate ideas. These all look like they’re parallel, right, harvesting –

[Simultaneous comments]

MARK:

Like pick communities has to go in front of this.

MARK:

Harvesting’s a little bit further down.

NOLAN:

Well it’s harvesting, right? This is harvesting new concepts from

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

[Simultaneous comments]

DREW:

I’d say that’s a generation game. So harvest is maybe a funny word for. It’s generate team concepts.

[General Conversation] [Laughter]

NOLAN:

The automated idea generation things are the easiest ones for, the first one you can do. You can do that before you have communities, right?

REED:

Or if you have like communities that don’t get funding. [Chuckles]

NOLAN:

Well the thing is, none of these actually have to be complete ideas. They’re all additive, at least right now.

[Simultaneous comments]

STRONG:

This whole system ought to be something that could feed back from all kinds of places, not just from the end and iterate through. And that’s a human judgment decision.

[General Conversation] [Laughter]

NOLAN:

Yeah, a block, big blocks will work nicely.

LOUIE:

The stacking ones have a natural connection, but most of the stuff can be done parallel, clustering.

[Inaudible comments]

NOLAN:

Steve, when I scribble this very broad category that’s evaluations, it both includes filtering, which is --

DREW:

Okay, so there’s some discriminating. That’s good.

[General Conversation] [Laughter]

MANSFIELD:

So if someone can start to be in charge of connections, which can be done with or with arrows or with markers

[General Conversation] [Laughter]

MANSFIELD:

The other thing is if there are subcategories that these are being organized under, it’d be good to I don't know if that exists or not. Maybe we’re just throwing them in.

[General conversation]

DREW:

Can you hear anything that’s been said?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BROWN:

Uh…

DREW:

You’re doing what you can.

BROWN:

I’m used to it. I’ve done this kind of stuff before so it gets kind of wacky. Yeah.

[General Conversation while doing task]

NOLAN:

from the communities, is that right, the identifying?

STRONG:

No, actually that’s more of an evaluation.

NOLAN:

So where does community?

McCORMICK:

Where’s the what?

UNKNOWN:

How do we identify new communities?

McCORMICK:

That’s probably going to come out of scenario analysis. Actually one of the ones I think is probably going to be the most critical in this whole thing is going to be the scenario analysis.

[Simultaneous comments]

NOLAN:

It would be one of the things.

[General Conversation]

LOUIE:

I think a long term forecast is you’re not --

STRONG:

Because you actually need signposts, you need something clear to vote on.

LOUIE:

The thing is, the vote, what you vote on is when a signpost or if it’s going to get hit at all.

STRONG:

Right, right.

LOUIE:

That’s where to expect --

NOLAN:

One of the things that your decision maker may be interested in is just keeping an eye on that, market and grows every single day.

LOUIE:

And you also pay for signals. In other words, if somebody delivers you a signal that’s useful, then you reward them.

[Inaudible comments]

STRONG:

And this actually generates more signals, the prediction market generates more signals to feed back into the system.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

NOLAN:

Okay, we’ve got a huge feedback loop going here. How are we doing?

[Inaudible comments]

MANSFIELD:

All right. Just set this up. We have about 7 minutes to put this together.

MARK:

Everything is in absolutely beautiful shape.

[Inaudible comments]

GOLDHAMMER:

Guys, so I can get your attention, if you can stop conversations just for a second. This is what we’re going to do. We’re going to -- why doesn't everyone come to this side of the room. Everyone come to this side of the room. We’re going to move around the room and see what everyone has created. We’ll have an opportunity for a quick sort of report out and feedback and then we’re going to go to a break. So if everyone can come to this side of the room.

[General conversation]

Team Activity: Identifying the Human and Technical Requirements

GOLDHAMMER:

Okay, so you’re supposed to be back with your teams, trying to figure out these additional requirements…

[General Conversation]

MARK:

So okay, cool. That works. My question is whether there’s also people who are specializing in doing this sort of analysis for a living, they may be some sort of consultants or something like that, they’ve got process.

KOH:

See, this can be automated, too, right?

UNKNOWN:

Or you find somebody who …

[Simultaneous comments]

McCORMICK:

‘Cause that’s what they specialize in.

DREW:

So you need both analysts as well as other analysts, plot.

[Simultaneous comments, banter]

NOLAN:

I think we’re weaker on our output side and we’ve got, we’re much stronger than the other teams

McCORMICK:

Also you could put this in front of that.

McCORMICK:

In front of that.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

NOLAN:

Yeah. This sort of analysis felt like it was very different, you have very, you know, more thought through the other team. The front end stuff felt in many ways, like ours were similarish to them. And the output, I think we’ve got a couple good examples in there, which was more comprehensive.

LOUIE:

And the way I look at it is that they’re more traditional, intelligence community process. This is like how do you take a big net and fish.

LOUIE:

Yeah, there’s a lot more of the fish in here. There’s a lot more fish and it’s kind of interesting because kind of different value propositions.

McCORMICK:

Well that one also assumes really big brains.

LOUIE:

But this requires a lot of expertise too. I mean, signpost generation and --

McCORMICK:

This is more process expertise than content expertise.

MARK:

It feels like that team may have a bunch more people who inside the intelligence community experience, which is thinner on this team. I don't know why they went for 2.0

LOUIE:

Hey, it’s very much crowd sourcey

DREW:

The contests look a lot more like the brand intelligence process.

UNKNOWN:

It’s because they have some web 2.0.

[Simultaneous comments] [Laughter]

MARK:

3.0. Isn’t it 3.0 when they get it right?

UNKNOWN:

That’s true.

McCORMICK:

Or was it 2.6?

MARK:

Remember the whole, you know, when your phone and they’re busy saying this is, you know, generation 2.6? You know, like how do you do generation 2.6? And I haven't heard that --

McCORMICK:

They’re about to do it again.

MARK:

I haven't heard that language recently. Okay, great.

[Simultaneous comments]

McCORMICK:

A lot of people can’t fully implement 4Gs in the rallies(?) and they’re doing 3.5.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

NOLAN:

Okay, this actually some of the most brilliant stuff ever struck out by the hand of man and woman. [General laughter] I’m saying that with humility.

[Inaudible comments]

MARK:

I mean, just to take that thread for a second, one of the hardest things is when we’re doing a to separate things like thought, volume in terms of noise from, you know, the little whisper of new ideas. I don't know what the system is for doing that but --

McCORMICK:

is a system. I think it’s How do you layer it so you can go from big picture things to lots of detailed data and backup

[Simultaneous comments]

McCORMICK:

In other words, you overwhelm somebody with too much information and you’ve got to be able to layer it in to be able to say all right, these are.

NOLAN:

That’s good 'cause I wouldn't even thinking of that. I was actually just trying to take a different blind story, which is some of these voices are going to be screaming and other voices are whispering and this huge flood of data that we’re collecting, you know, like how do we amplify the whispers and --

McCORMICK:

Especially important ones. It’s like they might be important but not urgent whereas other people are trying to create importance…

LOUIE:

But there’s an organization implied here that’s not really in here, which is there’s an organization implied that lives on top of this, who is querying, asking, poking, provoking, yeah, and concepts because this is kind of what I call driven focused, vision driven focused. In other words it starts with what does the future look like and then finding a way backwards into the technology versus other approaches which starts with here’s this technology. What does it mean for the future? So that we have a different kind of filtering mechanism that we say if it doesn't really affect any powerful vision, we’re probably not going to consider it.

McCORMICK:

So one of the key criteria’s about it is you have to have people who are extremely curious.

LOUIE:

Exactly.

McCORMICK:

Unbiased and extremely curious.

LOUIE:

Very.

NOLAN:

So let me also ask is there a way that the decision-maker, who right now is kind of at the end of the train, can say, “Gee, I, Mr. President, am wondering what about X, Y and Z. Can I somehow filter, dig in here and

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

see if there’s any ideas related to asteroids because I saw a movie and I was just wondering about asteroids.”

UNKNOWN:

Yeah, so what that – you would change it --

LOUIE:

Well the would've been put in the input so [General laughter]

NOLAN:

That one’s actually easier because that one would say demonic possession. Probably not in “What’s the technology of demonic possession? I’m the President. I want to know is there a way to kind of…”

LOUIE:

So the asteroid one is the easier one 'cause it starts with a vision. I see a vision of an asteroid hitting the Earth. What are we going to do about it? So then you kind of work your way backwards. The harder thing is here’s a technology discovery. What can happen?

McCORMICK:

Right. A technology looking for a home.

LOUIE:

This system may not be the best system for that. I mean, one of the other systems might be more interesting to use, which may be system one.

NOLAN:

So an example might be -- So when I was in college, high temperature, super connectivity, maybe going from what is 12 degrees C to 90 or 80 degrees, a huge breakthrough, they thought. Of course never did that. If that happens, it’s a technology disruption but what we’re interested in in many ways is the impact of the technology, not the technology itself.

LOUIE:

Yeah, exactly.

NOLAN:

So how would our system?

LOUIE:

You would look it up in the signpost maps to see if anybody had on a map --

NOLAN:

A story that was related –

LOUIE:

-- related that, where that thing could enable, right.

NOLAN:

Oh, so actually it would be the –

LOUIE:

It would be a signal.

NOLAN:

-- power transmission. That’s actually the thing that you might, might be critical in your story.

LOUIE:

Right, because it becomes the measure of interest and it becomes a signal and becomes a signpost because you can make a decision. If that comes true, now you have some choices you’re going to have to make. That’s all on this side. First you start on that side.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

[Simultaneous comments]

McCORMICK:

You could basically create from that the set of hypotheses and the signposts and then feed it back in there–

[Simultaneous comments]

LOUIE:

Exactly right.

McCORMICK:

-- and what you’re looking for.

LOUIE:

Right, exactly. Then you create new stories. Right, and that’s how it

NOLAN:

I feel like that we have to do the complete system. Is there anything else we can add at this point?

McCORMICK:

You know, actually there is one thing. What we don’t need is some kind of like a So it’s like all right, it’s, we’ve got the stuff that’s noise over here and then it goes into subjects of interest, then it moves into all right, these things actually have the potential of becoming really, really critical, which then, all right, these are the ones that actually really are critical. And we don’t have any like mechanism for that.

STRONG:

No, we do have one. It’s right here. It’s not explained even to us there. And in addition, if nothing else, the decision-maker or the senior manager of this organization might want to be able to say, “I need I need some information. How many ideas can be generated, where you guys are in the process, which things” -- This is the kind of thing that you can seed– I mean, you can change this depending on the usage.

[Simultaneous comments]

NOLAN:

That’s right. Impact depends on.

McCORMICK:

There’s the monitoring system for the process and then there’s the stuff that’s actually in it.

LOUIE:

So if you’re the Department of Defense --

McCORMICK:

Well I think you’re onto a good point because part of it’s also, like if you’re not getting a lot of change happening on the front end, you make your system starve.

LOUIE:

Yeah, that’s the only thing I’m interested in, which is --

McCORMICK:

When are people asleep at the wheel because it’s just too easy and they’re like –

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

STRONG:

Or when do I -- Maybe I can’t do all these things in parallel. Maybe it’s too research intensive. Maybe I have three or four of these and they’re working really well. When do I say I’m dropping this one, I’m picking up that one because I’m not getting the kind of fresh ideas I need?

LOUIE:

Well basically, you know, what’s producing these fresh new visions. Then you begin to track which ones – not just by quantity or by quality and diversity. So am I getting enough from, am I getting enough visions on how technology’s going to affect that? ‘Cause you might say no, I’m not getting…

STRONG:

What a great experiment to do, for us to compare harvesting with brainstorming.

LOUIE:

Yeah, yeah.

UNKNOWN:

I would love to see that comparison because there’d be two teams and you can have one on the harvesting –

LOUIE:

I’d be very interested in how some of these are going to --

McCORMICK:

And actually, the competitive component’s actually pretty good because if you have multiple teams going through each one of these and you’re pitting one against the other, you get lots of different perspectives, it kills the group think issues, etc.

LOUIE:

That’s right, that’s right. Because again, the measure isn’t accuracy of your predictions. The measure is breadth of your preparedness.

NOLAN:

And we may find that some of these evaluate, the evaluative processes are going to be responding more towards different types of users. So maybe in fact you’re like nothing seems to be --

McCORMICK:

Working on that one.

NOLAN:

-- working on this one, the chasm, we don’t have anything crossing the chasm, and we need to be asking ourselves is it because nothing’s crossing the chasm or is this an important kind of evaluative filter and we’re not generating enough ideas to hit that?

UNKNOWN:

Or you need a new filter.

UNKNOWN:

Yes.

STRONG:

That’s a way to generate signposts because I think that’s right -- You know, each of the models, the hype curve, the chasm, each of those templates is a template that can be used to crank out signposts, what we call candidate signposts. Then you evaluate different

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

McCORMICK:

And then ironically, the metaphor of analysis is a great way of actually picking up how people start changing and what they’re talking about, so you can then use it for sensing for like logs and stuff like that, these key words, you know, to figure out, okay, here’s a state change on that.

DREW:

That’s good 'cause one of the -- People, technology and partnership. Earlier Gilman said gee, one way to approach this is to take a norm, take the norm, cut down the 95% pieces and then throw away everything in the middle. Is there a way to do that?

McCORMICK:

Yeah. It’s actually a pretty easy one.

DREW:

It’s not the only way to do it but it’s --

McCORMICK:

I mean, actually it’s like, we did, I did analysis for Microsoft where basically we went on source gorge and basically did a full analysis of all the open source projects out there. You pull out the first like obvious ones and then everything that’s really the fertile grounds, everything in the next like 100, and –

DREW:

So where is that here? Where is that concept with blanking out everything in the middle? I mean, that’s a --

STRONG:

It feels like an important early filter though, before we spend a lot of time doing the detailed.

DREW:

So how – what people, technology and partnership would you need to do?

LOUIE:

It’s right in here.

NOLAN:

It’s actually -- You know what, I think where it plays in is it’s sort of another element of the scenario analysis in other respects.

STRONG:

Scenario is not just the narrative. There’s a lot of thinking that has to go into

McCORMICK:

In scenario you’re always talking about typically extremes. And then you say the reality is somewhere between all these different extremes that we’ve talked about, you know, both positive and negative.

STRONG:

I mean, some of it is explaining what the implication of a measure of interest is. And see that, you need all that and somebody has to write that down and explain it.

LOUIE:

Matter of fact, what you could do is you take your comparative analysis engine and apply it to the scenarios themselves. Has this scenario already been told by other prediction systems? And if the answer is yes --

McCORMICK:

One more day.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LOUIE:

Yeah, one more day, or maybe you don’t even worry about it. Just go on to the next,

[Simultaneous comments]

STRONG:

Actually, that’s really good what they were saying about. That is like something that can be done I’d love to do that and …

[Simultaneous conversations…]

[bell rings]

 

McCORMICK:

Look at it and say oh, okay, wait a minute. There’s a whole scenario here At other times you might just say okay, we’ll just take that out, you know. So I actually think, the nice thing about it is like there’s enough tools to automatically generate that stuff and then you need the human intelligence to say all right, you can logically take this out. Why don’t you just take that out.

STRONG:

And it’s enhanced human cramming the right kind of drugs and meth into these people so their brains are working [General laughter]

UNKNOWN:

That’s another one for the future.

[Simultaneous comments] [banter excluded]

STRONG:

And remember, they’re going to lose some other characteristics like ability to make friends.

UNKNOWN:

Yeah, you’ve been trying to play that for a while, haven't you?

LOUIE:

Yeah, works out here, Pac Bell. [Laughter]

[Banter excluded]

 

UNKNOWN:

Okay, so where are we putting this LSD factor.

UNKNOWN:

So any time we have procurements we need super humans.

[Simultaneous comments]

LOUIE:

Well you know, in some ways, you know, here we can do, we can extract tags. I mean, which one of these scenarios have tags showing?

LOUIE:

What new key words are showing?

LOUIE:

You know, what new concepts are being -- I mean, there’s a lot of stuff being worked on in some engines.

UNKNOWN:

The standard sort of 20 analysis.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LOUIE:

Yeah, exactly.

[Simultaneous comments]

LOUIE:

But I think we should note that, that there – I don't know how to do it but we need to write it on the board itself, listing why an organization who is going to be this process.

NOLAN:

I think it should be right here. Why don’t you write it there in a nice big box, every note is a macro story.

[Simultaneous comments]

NOLAN:

Write it on the paper.

UNKNOWN:

people who are managing the process. Yes.

[Simultaneous comments]

LOUIE:

And generally a lot of these boxes correspond to one person, a manager, not a team manager. One person. Some

UNKNOWN:

You know, actually I kind of like the idea of having competitions.

[Simultaneous conversations…no longer talking as a group…background noise loud]

LOUIE:

Feedback loops.

UNKNOWN

What I usually find for adoption for technology, it’s always you’ve got this great idea but there’s one little thing that’s missing. And once that happens, boom, it takes off like a rocket.

UNKNOWN:

Well it’s hard to identify that.

UNKNOWN:

Yes, it is.

UNKNOWN:

Because the web, remember the web, never thought anybody would hand code HTML, right? But yet everybody did, right?

UNKNOWN:

They did, that’s right.

UNKNOWN:

And so it’s not always easy to –

[Simultaneous comments]

UNKNOWN:

But also the thing about this. Look also what happened to the number of web pages as soon as you started creating the automated tools.

UNKNOWN:

Yeah, yeah, yeah. But I think the disruption happened before that though, right?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

UNKNOWN:

Sure, and some of them it does.

UNKNOWN:

It just happened better after. [Chuckles]

UNKNOWN:

I mean, some of it does happen but doesn't because like there’s some technologies that absolutely cannot be adopted, right, unless certain things are Whereas other ones, you’re right. It’s okay, you could sit there and go all right, there’s a basic enough level of code that I can kind of figure this out. ‘Cause let’s face it, there was already a skill base of people who you know, and HTML isn’t exactly that complicated of a programming.

UNKNOWN:

So another example, is actually domain naming, right?

UNKNOWN:

Yeah, yeah.

UNKNOWN:

I remember when I was and I was working on Internet technology and I thought, you know, my mother --

GOLDHAMMER:

Okay.

UNKNOWN:

-- will never understand any of this. There’s no way --

GOLDHAMMER:

Why don’t we come back.

GROUP 3

Group 3 Participants:

Moderator: Jesse Goldhammer

Harry Blount

Stewart Brand

Mark Culpepper

Danny Gray

Darrell Long

Ken Payne

Paul Saffo

Al Velosa

Stan Vonog

NRC Staff Member:

Sarah Lovell

Team Activity: Designing a Scanning System

GOLDHAMMER:

All right, so why don’t we sit here and let’s talk a little bit about what we’re going to try to do here.

GOLDHAMMER:

So, you know, this is a little bit of a tricky equation. We –

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

[Simultaneous comments]

GOLDHAMMER:

-- a little bit of a tricky equation 'cause we’ve got some design criteria that we identified that has disappeared but that we identified over --

BLOUNT:

They’re going to move us over there?

GOLDHAMMER:

We’re going to be moving over there. So our design criteria’s over there. So we have a set of design criteria that we came up with. We wanted it to be persistent, we wanted to identify anomalies, there was a third, which will come to mind in a moment, and we should talk about what’s the process, what’s the process we want to use, like how do we want to tackle this problem? There are a lot of different elements of a system. Where do we want to start?

BRAND:

In foundational terms, persistence, the idea there we’re cycling through the persistence might not be a bad way to approach the algorithm.

GOLDHAMMER:

Okay. So just to start from the beginning with the assumption that there’s some big loops, big persistent loops?

BRAND:

Or a sequence of revisitings or something – like iteration one, resolve to two, that kind of thing.

BLOUNT:

I think one of the questions is just composition of people involved 'cause bias came through very clearly, bias mitigation came through very clearly as an issue that we need to deal with. And so how do you thoughtfully construct the right or incent the right – people to participate including diversity

GOLDHAMMER:

Yeah. Well I think diversity, I mean, I think based on the conversation I heard earlier, making sure that we have mechanisms for incorporating lots of different opinions from around the world is pretty critical.

BLOUNT:

A basic one, I mean, a real, real basic one is the on-Net versus the off-Net population. We’re thinking about this from a data scraping, readily available model but, you know, the question I think of Ken is, you know, the guy that doesn't necessarily access the Internet causing disruption. So how do you account for that in terms of the concept model?

VELOSA:

Well the other thing though, this is an exceedingly over-educated group. I mean, I think the worst-case scenario is we have a Bachelor’s. So I think it’d be ideal to get --

BLOUNT:

Thank you. [Chuckles]

VELOSA:

Right. So, but the central point is we need some folks who are like only high school, right? I mean, it’d be great to get somebody who’s – and below, right? I mean, 'cause there’s plenty of folks --

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GOLDHAMMER:

Very much so.

VELOSA:

-- you know, and I’m talking not just over there, I’m talking U.S. You know, we need both, U.S. folks at that level, 'cause I mean, they can really contribute still but, you know….

GOLDHAMMER:

So just to give a practical spin on that, to borrow a page from Stewart’s playbook, like you can imagine doing a set of workshops in mega cities, and mega cities or something like that. You want poor, uneducated, highly entrepreneurial and inventive off the Net, although they probably are on the Net.

BRAND:

They’re probably more on the Net than we are.

GOLDHAMMER:

They probably are but, you know, I’ll get a couple of criterias in.

VELOSA:

Yeah, and actually the definition of “Net” is weird, depending on the population, because one of my cousins-in-law, he only does stuff on his phone, right? It means a high-school graduate. That’s it, right?

GOLDHAMMER:

Yep. Okay, so definitely a lot around diversity, a lot around persistence, we need feedback loops, we need young.

GRAY:

We also need, I think we also need to see the older folks as well. I hate to keep coming back to that but I think we do because I think they drive the wealth and they contribute to defining the need and they may or may not be on the Net.

PAYNE:

And then sometimes they evolve on the Net.

BRAND:

Many are. Some aren't.

PAYNE:

I mean, you know, a few years ago like hardly anybody over the age of 30, 35, was on Face Book. Now you’ve got forties and fifties or even older, you know. In fact the young kids are getting pissed off. [Chuckles]

BLOUNT:

Uh-huh. Cool.

PAYNE:

Yeah. And so, but that’s also another way to reach that crowd that wasn't available probably before.

GRAY:

I’ve heard the discussion, the dilemma of do I “friend” my grandfather?

VELOSA:

Yeah, I just recently friended my dad.

SAFFO:

Can we set up a competition? I mean, instead of scanning outwards --

VELOSA:

Oh, kind of like the End of Oil but --

SAFFO:

-- can we create an attraction.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GRAY:

That would be –

GOLDHAMMER:

Describe a little bit more what that looks like.

BLOUNT:

And it’ll look like an X-Prize?

SAFFO:

I don't know. Well, I mean, we have -- if it works for DARPA, it should work for us.

PAYNE:

We discussed that at the beginning of this committee.

GRAY:

Yeah, yeah.

PAYNE:

That was one of the earlier discussions, how we got people, incentivized people to participate.

VELOSA:

But actually, Paul, I would want to hear --

Payne:

But it didn't work out.

VELOSA:

-- to hear a question. I would expand on that a little more 'cause I mean, like that doesn't really seem --

SAFFO:

Well rather than a grand challenge, a whole bunch of little mini-challenges, depending on the question asked, you create a different fictitious organization to attract ideas. And you pay more attention to who brings -- you could use this a different way. So, one, you could use it to prequalify the folks who should be in the workshops.

GRAY:

So you have kind of that crowd, that crowd-source competition, and then within that --

SAFFO:

So you’re looking for --

GRAY:

-- you identify --

SAFFO:

-- you’re looking for elites within the crowd, which is where I think what we want are the hidden elites in the crowd.

GRAY:

And you don’t bias by educational factors or any other factor.

LONG:

I mean, you know, it’s my experience that’s kind of like [..?..] –

[Simultaneous comments]

SAFFO:

Yeah, intellect versus education. I’ll take intellect.

LONG:

-- so I have a strong bias towards undergrads. They actually, they’re smarter and they get their work done. [General laughter]

VELOSA:

If they’re into it, yeah.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GRAY:

If they follow the system instead of trying to fix it.

SAFFO:

I have one Ph.D. student who’s four years late on his paper. So to do that and then to also frame questions. I mean, because I really think this is about figuring out what the right question to ask is and then finding the right people to answer it. So can we create, you know, a honey pot that attracts that.

GRAY:

So maybe we use experts and generalists to identify areas and then use those areas to then ask the question of what are the questions in this area, what are the big --

SAFFO:

I don't know, if there’s a way. I mean, it kind of gets down to the specific, is there a way to make it even more concealed and, you know, think about a gaming company asking questions, giving out prizes, cell phones. That would bother Harry but – only Windows cell phones. Anyway, so that’s as far as the idea -- I mean, it’s an unformed thing. But what I’m trying to think about is rather than the mechanism that seeks and filters is to set up a lightening rod that attracts the ideas and people. It’s a lot easier to sit back and let them come to us.

SAFFO:

And one way to –

[Simultaneous comments]

BRAND:

We really need another anomaly detection. The peculiar thing about anomalies is you’ve got to know what everybody thinks that you'll find out what’s different than that, so some form of indicating the conventional wisdom or the official future, as we call it in the scenario business, even though it’s outside that.

VELOSA:

I think that should be a central foundation. We need to know what we know so we can start looking for what we don’t know that we don’t know.

SAFFO:

Well actually, I think it’s easier than that. You know, in our business looking for wildcards, surprises, whatever, it’s just cultivating a proper sense of the weird. So like the one I’ve been obsessing over for the last ten days was there was a helicopter, police helicopter shot down over a large city recently. It was not Bagdad and it was not Kabul.

GRAY:

It was L.A., wasn't it?

SAFFO:

Rio de Janeiro. And as far as they can tell it was – a army gun. You know, it’s one of them 50-caliber long gun things that the snipers use apparently. So my hypothesis is this is the new man pad. You don’t need a shoulder-mounted missile anymore. You just need a fine piece of military technology. And can you imagine the look of surprise on the face of the three police officers in that helicopter as they went down, because that wasn't in their contract, that they’d be shot down by

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

someone. But anyway, just an example. What I do when I look for – things that are weird just instantly get my attention and they keep rolling around. I actually think anomaly detection might be team based, a small team, choosing a network, pulling people in and then nose out, start asking questions.

GOLDHAMMER:

Yeah.

GRAY:

Would that be like specialty-based teams so you, so --

SAFFO:

You could do it as a specialty team. I think you could leverage the Web here and do -- You know, in Harry’s world, the cheesy part of Harry’s world that he doesn't really touch is –

BLOUNT:

[Laughter] Thank you, Paul.

SAFFO:

No, but the, you know, the stock gossip lists in Yahoo Finance, you know, all the folks who sit there obsessed about one dumb little company and every time it sneezes they all go, “What’s the stock going to do?” Is just have little -- if you could get groups of people having conversations around things and have some way to monitor it, pull out the stuff that seems weird.

BRAND:

Just make an entry for it on Wikipedia and assemble your group around the ongoing definition of the weirdness and you’ve got it.

SAFFO:

Brilliant. We can go home now.

VELOSA:

Actually that, but I mean, as one tool, that would actually be I think an important element to start doing things, just, you know, some sort of exceptions analysis on Wikipedia, you know, that multiple audiences. So there should be of course the internal version but Wikipedia --

BLOUNT:

Essentially it’s the anti-Google. Anomaly processing is the anti-Google, right? Because Google searches the most likely outcomes.

BLOUNT:

Right.

SAFFO:

Well, right, instead of the “I’m Feeling Lucky” button, it’s the “I’m Unlucky” button.

GRAY:

Yeah, “I’m Feeling Unlucky”.

BLOUNT:

Exactly. The “I’m Unlucky” button. [Chuckles]

BRAND:

It’s all Minority Report, isn’t it?

SAFFO:

One tool we really could use, I think something that would, especially for exceptions but across the board, is to factor into our system, when we actually get robust, image recognition and decoding so when we do a

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

visual search on the Web, searching images, my hunch is that’s a lot more useful to us than searching text.

BRAND:

You find amazing stuff with key words.

GRAY:

The AP has a section for the weird stories and, you know, or the strange stories, and I read that religiously because it’s – one, it’s funny and, two, it’s very informative because people come up with the strangest things to do this stuff and you usually find it there.

GOLDHAMMER:

So I’ve heard, I mean, just in terms of information that gets collected, I’ve heard Web-based information gets collected, I’ve heard sort of workshops, I’ve heard people inside, outside the United States, young people, people educated, not educated, poor, rich. Any sort of organizing thoughts on sort of what that comes down to in this first version of the system?

VELOSA:

Well Paul had a an idea.

SAFFO:

No, go ahead.

VELOSA:

Okay. Well one thing actually, maybe it’s a precursor to that or as a response to that, it sounds like the things we’ve been talking about are a little bit on the presence mode. Does this have to factor that into all those, the future mode as well as in applying, you know, a variety of levels of intelligence? And I don't know – and by essentially I mean actually just people, you know, eyes on the data stream. So one of the things to me is to make sure that as part of the persistence we actually have a cadre of folks just continuously looking at it and then tweaking it.

CULPEPPER:

Like a users group essentially.

[Simultaneous comments]

BRAND:

A users group is like that. I mean you!

VELOSA:

Yeah. You have to have a set of folks. And then have assorted users.

BLOUNT:

The only thing that worries me about the term “users”, it almost means you’re by definition including people that have a common interest, something which is more of a bias. So if the concept is a feedback loop, your original premise, then that makes sense to me. But then the question is how do you create the feedback loop that doesn't have some major inherent bias in the user groups?

VONOG:

But if you have a user group of people forecasting the future, then there is no bias. Like you could imagine people gathering just who are interested too.

GRAY:

Is there a disruptive technologies blog or, you know, or interactive space?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

VONOG:

Well, I mean, I could imagine like TED is sort of, they have TED events all over the world.

GOLDHAMMER:

There are.

BLOUNT:

I mean, you know, that’s the –

[Simultaneous comments]

GRAY:

-- [..?..], you know, or you could call it a futurist or something like that.

LONG:

I’m worried about creating a high Q echo chamber here.

BLOUNT:

Well, so here, but here’s where I’m going, is in the financial markets you have massive and immediate feedback, either amplifying or dampening, because as an analyst we put out a report that has a thesis on it and it’s a constructive narrative on why we think our thesis is right. And it’s amazing how quickly I got calls who said, "You’re full of garbage, basically, on this and here’s the reasons why." So essentially what you want is to essentially have something that stimulates that type of reaction so you get the feedback loop that’s immediate and fast. So the question is, is it almost speaks to needing some kind of broadcast of the narrative to a group that somehow or other is incented to respond and it doesn't have to be monetary incentive and maybe it goes to Paul’s comment about this grand challenge of being provocative in a heretic is a very creative way of getting a rapid, strong response that’s broad based.

SAFFO:

And let’s – I have a name for it. Let’s call it TIA, Total Information Awareness.

[General laughter]

 

GRAY:

But that’s interesting because by using that approach to stimulate the response, you could very quickly then pull out where are the pragmatists and where are the tails.

BLOUNT:

And if you iterate it, you create your own centralized mask that then causes further and further, you know, the ball dropping --

PAYNE:

So you need kind of like stewards to kind of put that provocative issue out there. And the good thing about that too is that when people really want to put you down they’ll try and bring the evidence in, you know, to prove their point. And then those folks who agree with you are going to bring their evidence in to prove your point. It may not be what you put down as part of your evidence. Maybe new people in there. So I think one of the things -- and we kind advocate, you know. I think stewards or something to that nature or some of those folks that kind of stoke the fire or keep the flames going are really important in that type of situation. And it’s not the answer but, you know, as far as you trying to get feedback on certain things.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

So what I’ve noticed here with what we’re saying is to get this reaction that you want, people to notice that, you have to challenge them, you have to be adversarial, right? If these guys agreed with your analysis they don’t call you and say, “I agree with your thesis. It’s wonderful.” Right?

GOLDHAMMER:

Not often. [Laughter]

LONG:

You get the phone call from the people that say you’re full of crap, right? And similarly, when you’re doing an analysis of technology or whatever, if you do things that confirm people’s biases and opinions they go, “Oh, yeah, that’s very nice, okay, good,” and then just continue their own way. But when you go and you gore their ox, you’re going to get a reaction. And I think that’s – you need to build something into the system that’s going, you want to, to get those people --

BRAND:

Yeah. It’s called top covers. [Laughter] [..?..]…

[simultaneous comments]

LONG:

You want those guys to react. You want them to react because otherwise people are going to be passive and just let things go. You know, we’re all very busy and if you confirm my biases I’m fine. Just let that go, right? It’s when you say something that says that I’m wrong, oh, boy, then that’s when I react.

PAYNE:

But it’s not a bad idea to have the things that you actually believe that may be institutional knowledge because you’ll get the outliers coming there too. I mean, how many times have you read an article that seems benign in the paper and then you read the blogs afterwards and there’s these people coming from way out like whoa, you know. And sometimes, so when you do like the party line or whatever, you know, you’ll get some antagonists as well and get thoughts that you didn't think about.

LONG:

But those guys usually get, in my experience, those guys get ignored, right?

PAYNE:

Right, but we don’t want to ignore them.

LONG:

It’s go along and get along.

PAYNE:

Right. But we don’t want to ignore them though. We want to pay attention to them.

LONG:

Right, but that’s not the way we do things now.

PAYNE:

We want to change that.

LONG:

Yeah, exactly.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BRAND:

What’s the role of demos here? I mean, I worked in the media lab for a while and deal with this demo We’re talking about technologies, new technologies, emerging technologies, potential threat aspect of these things. You know, red teams do one thing. But to actually – there’s lots of things it turns out you can’t make a demo of it. It’s probably not actually a threat or it can’t be until these other things that you’ve now identified come in to be part of the tool. So, you know, to succeed and fail in making demos of various things that one emerges with, if you get a successful demo, that you can take into somebody’s office and they know it’s impossible and you show them it works, then fuck, they’ve got to deal with that. So is demoing part of this process and, plus, it’d be fun.

LONG:

I think that’s a form of hypothesis testing, right?

BRAND:

It is a form of hypothesis.

SAFFO:

Well you know, in fact you could argue that the attack on 9/11 wasn't an attack, it was a demo.

GRAY:

Uh-huh.

SAFFO:

Because it really didn't kill that many people. But it was such a convincing demo they didn't have to do the real thing.

BRAND:

The problem was the design was, it was a one-off. [Chuckles] You only got to do the demo and nothing else.

SAFFO:

But it was a really good demo.

BRAND:

Yeah, great demo. Once.

CULPEPPER:

You know, as I listen to this I think about marketing and go-to-market type structures and a lot of times if you put out a press release in a business you do it, you’re doing it because it’s bait. What you really want is you want the media to respond to it and then carry your message, right? And you send these things out in varying kind of methodical cadences over time and you get, you start to get reflections back from the market, you know, from other people in the market, on the messages that you sent out. And that’s when you know that you’ve kind of hit a qualitative point of return like okay, you know, that message stuck and now let’s move on to the next message. And as I think about kind of what Darrell was saying on something provocative that evokes a response, it’s almost like the tool’s got to have some mechanism to send out these, to launch these little missiles out there that kind of immediately engender that kind of like bang, and develop it in such a way that it’s got a cadence to it over time so that you’re constantly testing, bouncing an idea, making sure that you’re getting that response, right? ‘Cause the response isn’t going to come at you immediately. Sometimes it takes months, you know, but it will come if it’s got the right content and message to it.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GOLDHAMMER:

One of the things that’s interesting about this part of the discussion is it seems like there’s almost a debate between to what extent do you intervene in the big system in which disruptive technologies are being conceived and developed, to what extent are you a passive observer of that system where you’re passively collecting information about it and seeing who’s doing what but you’re not really pinging the system. Those are very different models, it seems to me. I’ve for some instinctual reason feel like that pinging the system, it becomes very difficult to know whether you’re observing effects that are sort of native that you didn't have an impact on or whether you actually caused the thing -- You know, the Heisenberg’s principle: Are you causing the thing that you’re actually looking at.

BRAND:

Especially if you ping it in paranoid mode.

GOLDHAMMER:

Yes, especially if you ping it in paranoid mode.

CULPEPPER:

Well I think though that there’s, you know, from the standpoint of communication and messaging and how you got the responses back, typically they’re latent. They’re there. They’re just beneath the surface. And when you put something out there what happens is people respond to it because it’s already on their mind. It’s been something that’s been kind of like just kind of right beneath their skin. It’s been bothering them for a while and then, boom, that was the trigger that like got it out, you know. So you think about existing, you know, the market that I work in, the energy market, I mean, the thing I would want to find out for disruptive technologies is, you know, put out a message that says, you know, "Monopolies are a great thing and they really do great things for everybody and let me tell you how great it is," and see what response you get, 'cause I guarantee you’re going to get a response. And you'll probably hear it in very subtle ways. Sometimes you’ll hear it in very loud ways. But those are kind of the trigger points on the edges that you want to be able to look at and go okay, who reacted to that and why.

GRAY:

I think maybe what you want to look at is kind of a balance of the two, where you have part of the system that is simply asking the question and getting a – you know, in effect a list from the experts or the generalists and say here’s what we think are important. Then you take maybe some input from there, use it to seed, to try to find the outliers, you know, the --

CULPEPPER:

It’s a sonar, passive and active sonar.

GRAY:

Yeah, the people who are not responding in the normal are not part of the normal routes of generating that type of input. It’s funny because in the technology side you see this where you have engineering design competitions and they say, “We want this. Go forth and build one for us.” Like we saw a presentation about building a torpedo from, you know, common objects. And then the other side, you have like the Case-Coulter Initiative where – Case Western – where they just want innovative technologies. They don’t necessarily give a lot of -- Their

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

guidance says I want an innovative technology that’s going to be, you know, good for people and have a clinical use. And that’s it. And then they get, you know, 25 ideas and then they go through those and find out what’s viable. Now you can make an argument that from there the tech transfer system fails but at least from the innovation side, they do both pieces. They have directed innovation and then they have kind of open-ended innovation.

VELOSA:

And I really like actually what you guys have been saying so far because in terms of going back to the weak signals, it’s one of those where, you know, it’d be really interesting to do that kind of active/passive sonar. I think that’s a great analogy to be using. But the other thing is to actually then do these competitions and, for example, one of the things would be to set up so you get different educational levels. For example, get -- I mean, like all undergrads usually in orientation have to – in engineering school, sorry – have to go build, you know, drop and egg without breaking it, right, you know, the usual kind of thing. But maybe you’d do something in particular towards find some high schools or alternatively, get new recruits in the Army or something, you know, get a population with just high school, a population with college, you know, and try in different locations with some of these signal kind of sensors. So you do both the competition as well as the passive/active sonar.

GRAY:

It’s interesting because the systems biology world came up with this international systems biology competition and their idea was, you know, you have to come up and do these things. The problem was to some degree they were snobs about it and they opened it up to university campuses. Well, what about people who do this in their garage that are 13? You know, that’s who we want to talk to. We want to talk to that 13-year-old that made it in his garage not the 25-year-old graduate student who’s well on his way to a Ph.D.

CULPEPPER:

You know, one of the things that I’ve noticed about disruptive technologies and people who are involved in it is they have – they’re disruptive and that’s the nature of who they are. They like to be disruptive. And if you put a message out, it is going to come back at you, you know.

 

[laughter]

PAYNE:

Yeah, but sometimes, you know, there’s something to be said about naïveté. And that’s the other people that are disruptive, that they don’t know that they can’t do that.

GRAY:

Absolutely.

PAYNE:

They don’t know that it’s not possible and so they move right ahead because nothing is encumbering them, you know, intellectually from stopping -- you know, Al brought up a good point. You know, if you have too much education, you learn too much about what’s not supposed to be and what you can’t do.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BLOUNT:

We run something every year at the Tech Museum called the Tech Challenge for grades 4 through 12 and it’s a real-world problem. Last year it was landing a scientific package in a volcano. This year it’s about space jumping. And we also give out an award, by the way, for the most spectacular failure. We want to encourage the failures. But if you go and observe this, the creativity for opportunities to disrupt and these common thoughts., is actually staggering.

VONOG:

Uh-huh. So maybe it’s an input from the system because there are lots of these competitions going on. Those that I have participated called the Imagine Cup, which is 300,000 students all over the world for Microsoft. You just take a brochure of worldwide finals, it’s like – and kind of see what – flip through countries and you can see like all the problems that are interesting. And the topic there is like eight problems that UNESCO thinks are most important or like imagine worldwide technology just bounded. And I’m sure there is like for biotech, or like children’s things, so there is a compilation of a list of those competitions.

SAFFO:

And for small scholarship awards.

VELOSA:

I’m sorry. Keep that -- Can you just make sure you write down that one of the things we should --

GOLDHAMMER:

Competitions.

VELOSA:

-- you should do is have them, yeah, survey all competitions to see what they are doing.

[Simultaneous comments]

GOLDHAMMER:

There’s a competition data set.

VELOSA:

Oh, there is?

VONOG:

Yeah, and it’s free, like --

GOLDHAMMER:

It’s a great one.

GRAY:

Well I mean, there’s a -- The Learning Channel actually has something for grade schoolers that I’ve been interested in 'cause my daughter’s five and a half.

VONOG:

And maybe you have a scout, like in the Imagine Cup they have demos, worldwide finals and all.

GOLDHAMMER:

Yeah. There are two sides to this and I feel like the conversation – and I think we’ve touched on one side of it, which is technology creation, competitions that are getting people to do things in new ways. What about the use and adaptation part of the story?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

VONOG:

Well, in those competitions it's part creation, part user adaptation. There is, people are doing all kinds of things.

GOLDHAMMER:

Yep, that’s fair.

SAFFO:

It’s a narrow thing but it would be interesting to broaden out, is follow volume and pricing of key technologies on eBay.

GOLDHAMMER:

Yeah, yeah.

UNKNOWN:

That’s a good idea.

SAFFO:

So last two years I’ve been following the price of thermo cyclers on eBay as an indicator about bio-hackers at home.

BRAND:

Oh, that’s interesting.

VONOG:

Well that’s kind of – Gilman was talking about measurements like bandwidth or --

VELOSA:

There’s a way to track it?

SAFFO:

Oh, you just, I just go up there about once a week and I -- There are not a lot of thermo cyclers so…

VELOSA:

Oh, okay. Okay.

GRAY:

But you’ve got actually – that’s a really good point because as labs are closing down all over and this stuff is hitting the market – again it goes back to the waste stream. And this is a biological waste stream of stuff that’s not 25 years old, it’s five years old or less.

SAFFO:

And you can take it a step further. It never really occurred to me to send an email to somebody who just bought a thermo cycler but it would be really interesting to follow up.

GOLDHAMMER:

Yeah, with the guys in the suits who knock on the door. [Knocking noise] “Scuse me.”

[Simultaneous comments]

GOLDHAMMER:

“We’re here from the government, we’re here to help.”

GRAY:

Yeah. [General laughter]

SAFFO:

Please. That happened to me when I was 12 years old and I still remember it.

CULPEPPER:

Well I think that’s exactly the kind of passive structure that’s really strong, right?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GRAY:

Right.

CULPEPPER:

It’s very scalable. I mean, like that kind of system is like you can ramp that up very, very quickly, you know.

GRAY:

I’ve been following the do-it-yourself biology blog.

BRAND:

Good. What do you find there?

GRAY:

It’s interesting that people are doing just what we’re talking about. Somebody posted instructions on how to make a biological fume hood. You know, somebody else said, "Here’s how you make –" you know, some other piece of equipment and it’s being made out of everyday materials.

BRAND:

I know the guys behind that blog and it’s pretty interesting stuff.

GRAY:

Yeah, it’s interesting but it’s also a tad bit scary, you know.

BRAND:

Well, what they’re doing is basically adding to what MIT is doing with the iTune meetings and so on and saying, "Look, we’re leaving the amateurs out of this and the amateurs are going to be in it anyway so let’s work with them."

GRAY:

Uh-huh. Well, and as I was talking with Ken, one of the things that you’re getting now is a bunch of disgruntled post-docs who are no longer in the academic hierarchy, you know.

BRAND:

Is that right? Oooh.

GRAY:

And so what do you do when you go and become a drywaller or you go and become, you know, a business guy because there’s no career path for you in academia anymore but you still love science? There’s where you’re tracking–

[Simultaneous comments]

BRAND:

Okay, so the disgruntled post-docs is a population of great interest to this group, it sounds like, and they’re findable, I assume.

GRAY:

Well they’ve, they have, in the last five years you’ve trained about 25 times more post-docs than the system could feasibly use.

BRAND:

How many of those are going to a home country which is not the U.S.?

GRAY:

Not a lot.

BRAND:

Okay.

GOLDHAMMER:

So lots of different classes. What about information processing? So we talked a bunch about information that needs to be collected, different

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

ways in which it gets collected. I think we can kind of capture some of that stuff on the table. How do you make sense of it? Is it, you know, is there a central, a coordinating function to this whole thing which has kind of an analytic component to it? Are there different analytic teams? Like who’s looking at this stuff and saying like "That matters, that doesn't matter, that matters, that doesn't matter"?

PAYNE:

Well, and I’m not sure how a basic system is but some of the folks who’ve been on a committee – talk about building something about defense technology in one system. And part of that is you have 20 technologies to work in, 20 different disciplines. And for our office they’re supposed to be the ones to do that. How we would do that – and we would like to do that for the, you know, DOD, DDR&E because we have the same interests, you know, from the intel perspective we’re just worried about innovation really and use, you know, from that threat. But our interests are similar, more out to the, you know, Department of Defense, DDR&E( folks who are doing the research and engineering). And so it kind of can cross both ways, you know. That’s how we do it but that’s not --

GOLDHAMMER:

Right.

PAYNE:

But I kind of want that problem solved within this group without that information.

GOLDHAMMER:

Okay. Forget you just heard that.

PAYNE:

No, but, well some of the folks, the committee members have heard that and so I kind of would like to see that resolved outside of that, where you would develop a system that does that. I mean, maybe you do have some technology stewards. Maybe it’s not 20, but where do you get them and how do they operate within that, the hierarchy?

BRAND:

That’s one example. What are other exemplars of this kind of process?

VELOSA:

Well, but the other thing though is it’s technology, right? I mean, what about use 'cause to some extent it’s actually – that’s more relevant 'cause the IED was not a technology, it was a use. So it’s just like is there a way to have use or --

GRAY:

I think you might be able to address that by targeted competitions.

VELOSA:

Oh, through the …

GRAY:

I think if you were to challenge people, you know, kind of like the Navy did, well make a torpedo out of common things. Well how did they do it? What did they do it with? And so you simply say, you know, given what you can scrounge or buy for and maybe put a dollar on it, say you can't spend more than $200 to do something or a hundred dollars to do something and then kind of, you know, address, you know, give them targets. And you could have either a physical contest and they have to

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

build it or they just frame it out and send you a sketch on it, you know, and their idea. I mean, it’s one or the other.

BLOUNT:

So it’s like a design challenge, learning it. You could almost use Amazon’s mechanical platform to facilitate something like that.

GOLDHAMMER:

In my experience I find creating almost just sort of a mini version of the kind of diversity perspective that you’re hoping to see globally and doing that with your analytic team, it could be quite powerful. So you’ve got, you know, you’ve got the systems biologist, you’ve got political scientists, you’ve got intel people, you’ve got basically a huge cross-section of different people who are trained in different ways but the basic purpose is to look at the data that’s being collected, a lot of the data that we’ve just discussed, and figure out what it means and why it matters.

LONG:

You guys just said something that actually worried me a little bit and that is we’ll have these competitions and we’ll get people to try stuff. A lot of the stuff I’d rather that we didn't give them the idea.

GRAY:

There’s an argument that way.

LONG:

So there was this guy in New Zealand --

GRAY:

Yes, there’s an argument that way as well.

LONG:

-- a few years ago that – he was a very interesting guy. He was all into pulse jets and he was going to make a cruise missile for $10,000. Fortunately, I think he was discouraged from this, okay?

[laughter]

 

LONG:

But, you know, I would rather not encourage these guys to do this. What I want to do --

PAYNE:

But --

LONG:

Hold on a second. I need to figure out if people are going to do this but I don’t want to seed ideas to adversaries --

PAYNE:

Right.

LONG:

-- is what I’m a little bit worried about. So let’s have a cruise missile competition, shall we? You have $500 to build a cruise missile. I’d rather not do that.

VELOSA: But you could have the competition inside, you know. I mean, just have --

GRAY:

Yeah, that’s fine. The Academies are wonderful for those kinds of things.

VELOSA:

Well, but my point is actually get away from the Academies. Go to the enlisted folks 'cause again, you’re -

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GRAY:

Yeah, the enlisted guys do great stuff. Like for example, one of the IED defeat things was done by an enlisted or I think it was a sergeant, right? And what it was, it was a heated plate on the end of a long arm.

BLOUNT:

You know, the only thing I worry about with that though is maybe if we don’t put it out there it will go away and --

SAFFO:

Well the people to consult. This is a problem that’s faced by police departments all the time. In fact, there’s a case moving up to the Supreme Court about entrapment as we speak. So I think it’s just a design detail that one has to come up with criteria about what are the safe and proper things to encourage. I mean, forget cultivating terrorists. I mean, this really is the TIA outcome where you don’t want to create something that’s a political hot potato. But that can be a design element, that’s a design problem. So if I put a box in here, you know, there’s a risk/benefits test to everything that’s done.

PAYNE:

I think some of the things too when you look at this, as you gather information, whichever way you gather it, you know, is -- And Harry, how would you sort through that to figure out what to invest in? How would you sort through that data and say this is a disruptive technology that I want to get ahead of and I want to be the first or one of the earliest ones to put my money towards that? I mean, how would you sort that data out to make that decision?

GOLDHAMMER:

That’s a key question.

GRAY:

I think Stan had a good idea earlier and that was I think we’ve identified kind of this top, at this point we’ve identified kind of the top two pieces to get things to then be in a list of actionable or analyzable topics now and maybe we’ve identified, you know, weak signals, strong signals. So now what does that analytical piece look like? Is it a, you know, a science component, is it a finance component, is it a political science component, you know, do we have pieces that feed this and is it, you know, rank, you know, front ranking, is it a feasibility ranking or how do we go through that?

BLOUNT:

Uh-huh. In answer to your question directly, this is a true case example. When I was covering technologies, hard drives, I put out a thesis. I had the call on hard drives for years and all of a sudden I got this phone call saying, “If you’re not thinking about this, you’re nuts, you’re missing the whole point.” And the word was residuum, which was a word I hadn't heard since high school physics. That was a weak signal and all of a sudden I had to make a bunch of phone calls. And it quickly became apparent that -- and so one of the key criteria here is I knew who my customer base was and what was important in terms of impact. A few phone calls it showed up that the company that had been thoughtful about securing the control over residuum had a significant margin advantage, profit margin advantage for a number of years. They controlled the supply chain. And then I was able as an expert to assess

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

the impact and go back to my stakeholders and say, “We need to buy the stock because…” So I think as a design criteria it is, once you have the signal filtering mechanism, is the people that understand the stakeholders’ impact lens or filter, whatever it is.

SAFFO:

Who? Go ahead. Who was the company?

BLOUNT:

Seagate.

SAFFO:

It was Seagate.

BLOUNT:

It was a byproduct of the gold manufacturing process. Who would have thought? I mean, there’s like three people in the world that knew --

GOLDHAMMER:

But that was also a trigger for you to go ahead and make those phone calls.

BLOUNT:

It was the weak signal. It was a word I hadn't heard in twenty years.

GOLDHAMMER:

But the question is how do you design a system so that Harry Blount is the one who’s looking at the weak signal and actually recognizes it as a weak signal as opposed to me, or any one else.

[Simultaneous comments]

SAFFO:

Harry is a spider at the center of this vast global Web and he feels the little vibrations coming to --

VELOSA:

Yeah, and part of the point you said has to emphasized because it’s Harry Blount who people pay attention to because he’s demonstrated relevance before, right?

GOLDHAMMER:

That’s right.

VELOSA:

You have to have folks that are, can take a weak signal and then get listened to as they come out --

GOLDHAMMER:

Well here’s a totally radical proposition, just to follow on that point, which is what if V.2 of the systems is not about identifying disruptive technologies? What’s the low, I mean, is there the so-called low-hanging fruit in the disruptive technologies space and is the first version of a system just simply optimized for identifying the disruptions in X so that you can establish that credibility and you can actually start building out from disruptions in hard drives, disruptions in telecommunication devices, disruptions in something actually quite specific, and then build out from there? You actually develop the model not around broad disruptive technologies but something very specific.

BRAND:

That would take very intelligent systems growth because typically when something becomes expert something early on, that sticks with -- It took a long time before there were any biologists on the Jasons, as I

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

remember, going all the way back in its history. And that lapse like showed. Any of these things, you want the work-around. One I’m trying to think of a work-around on is best way to get good analysis and good judgment is to have a brilliant fucking leader and, you know, you go through the process. Usually you’ve got a brilliant fucking leader and you keep him or her and all the rest of it. And if they go away – [loud laughter in background]…process. Great. But all brilliant fucking leaders have blind spots which can kill you, depending on how -- they’re really, really brilliant, they’re totally persuasive, they’ve got all this fantastic record to show you how great they are and everybody’s in awe, you know. "I’m a Rick Oliver et al" or whoever, but there’s blind spots. So here’s the design problem. With the collusion perhaps of this brilliant fucking leader, do we have blind spot analysis and action so that it doesn't become the thing that trips you up? I don't know the answer to that but it’s what I would want to do with something that depended on good judgment.

GRAY:

Maybe we need to identify the, you know, the professional communicator. Because we’ve talked a lot about how to communicate this to the eventual customer and knowing your customer. So maybe it’s we have an analytical piece that then feeds this front person who then has the credibility to bring the information out. And, you know, if you have, say, five or six functional areas that are, you know, we came up with – we identified some signals, we identified some populations and got some big questions and we brought it down and this group of five or six areas worked on it and then you turn around and you feed it forward to somebody who is the presenter. And that’s, and to some degree you might want them – I know this sounds funny, but you might want them blinded to the process of how they got that information. Because one thing that might kill you is when the questions are being asked, you know, I don’t want to use culpable deniability but I’ll just say it might not be a bad idea to keep what happens behind the curtain behind the curtain, you know.

BLOUNT:

I guess the question is, from a design standpoint, do we start by saying with the stakeholders and whoever the platform is what is -- an assessment algorithm, basically, what is the threshold level of, if it hits this many people or it hits this type of criteria, you know, it tends to resurface much more readily. I’m putting that out there as a question. In my world it was stock impact, you know, market cap – sorry, Paul. What’s a better word for that, market, market effects? Sizing market effects. And so in your case, you know, if I asked you --

PAYNE:

Once we see the effects, it’s too late. [Laughter]

BLOUNT:

Yeah, but there’s a high level question, which is how do you – what’s important to you in really having it in a thoughtful threshold so that when those weak signals come up there’s some kind of --

LONG:

No, offense, guys, but I think we’re becoming a little too reactive here. We’re talking about, you know, deriving things from signals and things

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

like that and I think then we’re behind the curve on this. You know, Stan brought this up and other people have brought this up, that we need hypothesis generation for something to happen, right? People need to be coming up with -- well, you know, you’re talking about technologies, right? What happens if a disruptive technology happens in hard drives or communication or whatever? That’s very nice and I’m very happy for that technology. But as I said earlier, it’s the use of that technology, putting together a couple of things that make the real big transition. And we want to be ahead of that transition. We don’t want to notice that transition as it happens. We want somebody to have said, "Well, you know, if – I want to do this, you know, some thing," right, and could we do that, make this hypothesis? And then you have other people that are gathering data and looking at this and saying it’s possible for this to happen but in order for this to happen these things have to happen and then you watch those things.

SAFFO:

So we have – at Longnow there’s something that kind of works here. It’s I think – it’s a variation on prediction markets but I actually don’t like prediction markets. I think the whole idea is cockeyed. It’s called long bets, where, you know, if just one person wants to do it it’s a prediction but if you have two people with opposing points of view, they go on record and -- Well, you should describe it.

BRAND:

All I wanted was accountable predictions in the world 'cause I’m tired of going to all these conferences where guys wave their hands and say in ten years’ time hard drives will, whatever the hell it is, and everything, and –

VONOG:

Gilman has a portfolio company, by the way, who --

BRAND:

I’m sorry?

VONOG:

Gilman has a portfolio company, by the way, who sold accountable predictions inside like big organizations so…

BRAND:

Right.

LONG:

What is an accountable prediction?

BRAND:

An accountable prediction is a person’s name on a falsifiable statement about something that will or will not happen by a certain time in the future and their argument --

LONG:

And their ranking goes up or down based on their ?

BLOUNT:

-- their argument about – their theory of the world that makes that particular thing come to pass or not come to pass. And the accountability is that this is kept online, it’s voted on, it’s argued about and then the time comes to pass and it happened or didn't happen. That’s all.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

CULPEPPER:

Yeah. No, I think that’s one of the best tools out there like for exactly what you’re talking about.

BRAND:

Sorry?

CULPEPPER:

I think it’s one of the best tools out there that I’ve seen, actually.

BRAND:

Thank you.

CULPEPPER:

That long bets are exactly –

BRAND:

Use it more.

CULPEPPER:

That’s it.

BRAND:

Martin Reese has one and nobody’s taking the other side. I think he said there’ll be a million people that die from an instance of bio-terror or bio-error by 2030 or something like that.

VONOG:

I had a small point about the anomaly detection. So just I think it’s useful to assume it's a solvable problem within like short period of time because, first, like an anomaly is an outlier in some set of coordinates. And second, I know like a few companies who are working on kind of that stuff, like -- And third, like just as I’m thinking, if I could design the system like you will get sort of like startups tech crunch. You have this Web, Web, Web, Web, Web application or tech crunch like you have like 50 Web application and then you have like one sort of iPhone controller thing. So that’s like already some kind of an outlier and it’s not like a prediction but you could have some automatic prediction built pretty easily. And a second point about – so Gilman’s company, they are doing – So the guy, the founder, worked at EA, it’s a big company, and they kind of making this prediction when product is going to ship and how much users will use it. And they’re wrong all the time. And he found out like if you talk on a soccer field to different people you can be so much more accurate in those predictions. So he left EA and started this company where like he sells to enterprise and they make bets and that kind of system makes much better prediction. They’re 60% more. But that’s like short-term thing, not long term, not ten year.

PAYNE:

Well, Stan, when you started the first(?) company --

GOLDHAMMER:

Well let me suggest that we --

PAYNE:

No, I was asking Stan, like, you know, he started his first company and how old were you when you first started your first company?

VONOG:

23.

PAYNE:

Yeah, and so what was your thought process to say here’s what I want to do. I want to start this company to do X. What drove you to that.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

VONOG:

Yeah, in my case it was kind of -- So I was participating in these competitions and then in 2005 we won with this music thing when we imagined worldwide technology causes boundaries between people and we want it. And like why not start the company?

GOLDHAMMER:

Why don’t we get up and move over to our table and get our blood circulating.

[Move to project table]

[General conversation]

PAYNE:

Sometimes it’s the question that’s asked. You know, we’re asking a really big question. Hey, solve all these things, right, where we can say, "Hey, how would we find out if somebody is developing X?" and then we could go right down sat, okay? We could go right down the slot.

GOLDHAMMER:

Word for word.

PAYNE:

And so we realize we’re asking this difficult question, we do, and so maybe we can narrow it down to technology, I don't know. I don't know. physical science or basic to whatever. That’s why you get so many divergent things like here’s where we are. We know we’re doing that.

GOLDHAMMER:

You know, whether it’s one technology or a bunch of technologies, it’s primarily a systems problem and the thing about systems problems is there are a million different places you can cut into a systems problem and you’re looking at the system from every different angle. So we talk about collection, we talk about analysis, we talk about different ways to support. All that’s required. But what’s hard to do with in groups like this is how to do, sort of think through systematically like what are the actual, what are the priorities. But a lot of the conversation…

[Simultaneous comments]

PAYNE:

And don’t concentrate on intelligence. We’re at the very tail, we’re at the very tail end. All these innovations have happened by the time we really, all these technologies and science have happened by the time we’re willing to something. ‘Cause now we’re concerned about, you know, why would somebody want to do something against us and then, you know, what will be their expertise or what do they have available to them, all these other things. So we’re looking at other things by the time it gets to us technology-wise. And so that’s why from the DDR&E's perspective we want to know this technology so we can get out in front of it so we can develop things. And so -- And that’s why I’m not -- That word about use drives

[Simultaneous comments]

PAYNE:

Right, and identifying, you know, what’s going to drive technology and those things, and then we get -- it’s funny. My boss has this thing like a -

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

- it’s like a, he calls it the heater scale. You know, those old heater and air conditioning things?

GOLDHAMMER:

Yeah.

PAYNE:

You know, where it would be like blue at the beginning and then as you got towards the end it looked like more red. And so blue is the DOD and we’re the red. And so like at the beginning of the phase over here on the left-hand side you’ve got all this blue and it’s almost all blue the whole way and it slightly starts becoming red, red, red, and then down towards the end of that is the red part. That’s where we get involved. So we’re really trying to look at that first part of the blue part and then that’s why it’s open. I mean, by the time –

[Simultaneous comments]

GOLDHAMMER:

So let me make a suggestion. I think this is the kind of thing that as much as I would love for it to be one single process, I think we should have a parallel process. Otherwise I don’t think we’re going -- I think we need to put a couple different stakes in the ground. And before we start writing up big things we probably should write on little things. So let me --We’ve had a bunch of different conversations at the table about different elements of the system. What I’d like us to do is in a small group, a couple people, take some Post-It notes and start blocking out like how, you know, if you’re interested in, if you think competition is really important from one part of this, either for collecting information or for processing information, what does that look like? If you think that’s some other element that we’ve talked about, so, you know, like long bets is an important part of the system for surfacing the key issues that we need to look at, how does that look like? Who’s making the decision about what the bets are, what is the process for making those bets, where does that output go? Let’s sort of break this up into a couple of different pieces and then see if we can put it together. And I’m also open to other suggestions but it just seems to me that because it’s a systems problem, everywhere we cut at this it just gets massive quickly.

UNKNOWN:

I’m glad you said that 'cause it is a big problem.

GOLDHAMMER:

It’s a big massive thorny systems problem. So if there are no alternate suggestions, let me break pieces off and feel free to start doing some designing. Let’s see if we can -- It might be widgets that we end up with and it might be a system we end up. We’ll see. All right.

[Simultaneous comments]

GOLDHAMMER:

And can I make also another suggestion? Let’s just focus for the next 15 minutes or so on collections, different kinds of collections over on this side. This is going to be very hard to take that apart. I just apologize in advance.

UNKNOWN:

I’m assuming you mean both passive and active?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GOLDHAMMER:

Both passive and active, exactly.

VELOSA:

So you want data collection on any particular area or --

GOLDHAMMER:

Yeah, well let’s just --

GOLDHAMMER:

-- on this side of the table?

GOLDHAMMER:

On this side of the table let’s do collections. We’ll do analysis, we’ll do narrative, we’ll do reporting.

VELOSA:

And I’m sorry, you want us to do big fonts?

GOLDHAMMER:

Big fonts. Yeah, so let’s just figure out what are the elements of the system. There are pens here if you guys want [..?..].

BLOUNT:

So we just throw stuff down? Someone else is going to organize them?

GOLDHAMMER:

Yeah, let’s throw stuff down then we can talk about it. How does that happen? Who’s doing it? That’s what we want. The concept’s great. How does it actually happen?

LONG:

How about this?

LONG:

We talked about this.

GOLDHAMMER:

Can you block it out?

LONG:

Can I block it? I have no idea what you’re talking about.

GOLDHAMMER:

In other words, can you show me what are the steps in the pro-, whose --is some group generating questions, some group generating hypotheses?

LONG:

Yeah.

GOLDHAMMER:

Is it five groups, is it three groups?

LONG:

How much money do you have?

GOLDHAMMER:

You know, assume for the time being that, you know, it’s finite.

LONG:

Yeah, okay.

GOLDHAMMER:

Okay?

LONG:

All right. We’ll scrap that.

CULPEPPER:

So media tracking is basically, there are a number, not to be too technology centric here, but there are a number of systems out there that allow you to track qualitative and quantitative data and that’s what that’s

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

about. That’s basically a passive monitoring system, put it that way. One angle

[Simultaneous comments]

GOLDHAMMER:

And is that purely computational?

BLOUNT:

Yeah, that’s literally looking at press releases, looking at what goes into a different media story, anything basically that has to do with it gets out into the public world. You know, odd stories, anything like that. Right? It’s a purely quantitative passive collection.

GOLDHAMMER:

All right. And capability tracking? This is -- tell me more about that.

BLOUNT:

So basic things like, does somebody have the materials, the education, the money.

GOLDHAMMER:

Okay, okay. Computational and human?

BLOUNT:

Or it just could be access to raw materials.

GOLDHAMMER:

Access to raw materials.

BLOUNT:

So part of it is if you have a thesis, what are the requirements that are going to make that happen.

GOLDHAMMER:

Okay. And money flows?

BLOUNT:

Same thing. Follow the money.

GOLDHAMMER:

Follow the money. Who’s buying stuff on eBay, money flows?

CULPEPPER:

So you include in that, Harry, kind of P&L, just any sort of money flows, capital flows, or down to a P&L at a company level?

BLOUNT:

I think it’s actually maybe even more basic than that, is following, like research grant flows, how -- You know, the first time you see a new flow of money into an area, that’s an interesting data point. So it’s probably, you probably want to -- what’s more interesting is a new flow and then a rate of change in flows.

BRAND:

And watch for a language, new terminology.

BLOUNT:

Like the residuum within my lexicon.

BRAND:

When hackers came up with a hackers’ dictionary that was new

CULPEPPER:

That’s kind of what I’m talking when I say message program response. That’s really exactly that kind of thought.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GOLDHAMMER:

Enthusiast tracking. I like, what I like about what we’re doing here is it’s not just tracking information but actually specific things that are getting tracked or enthusiasts, media, capability, the money. Say more about some of these, story generation, collaboration tools.

VONOG:

So story generation was this direction of thought that when you start with a story and then develop data around it and think about the stakeholders first and how you sell it.

GRAY:

Analysis.

VONOG:

So then story generation comes. And collaboration tool, people who build networks, that’s kind of when we talked about data overloads, how people deal with data overload now. You have like Twitter, Face Book, you just have like trusted people and you hide those which are noisy and you get less people. So this kind of social network tools to help get through that noise and overload. And data collection’s just like I guess they can work

GOLDHAMMER:

It sort of generically describes a lot of the other stuff.

VONOG:

Most automatic stuff.

GOLDHAMMER:

Okay. Irritators' diffusion rates. Can you say more?

BLOUNT:

So diffusion rates is if you have disruption, a disruptive event might be easily diffusible across the globe or it might be very hard to diffuse it across the globe. It kind of gets into not only, it gets into not only collection but an impact. And irritators, this tends to be your outliers where you get the strongest reaction to polarization, what causes the polarization of event.

GOLDHAMMER:

Okay. As you’re looking at this, I’m also interested if you see ways to organize it, whether they’re things you would do, whether there’s priorities or the things you do first, things you do second, things you do third. And I’m interested in not actually – it’s both adding to the list but also whether there’s a way, whether you see ways in which we can organize the different things that we need to look at and to gather.

LONG:

So you’ve driven the stake in the ground here --

GOLDHAMMER:

Yes.

LONG:

-- in collections. Not everything here is a collection.

GOLDHAMMER:

Okay.

LONG:

Right? And I think for me this is very constraining actually, to say okay, we’re doing collections. But I, you know, we say this is a system problem, right, and there’s an overall system architecture and we’re

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

trying to look at this one little piece, right? And I put something down and you pooh-poohed it so I tore it up and it’s gone.

GOLDHAMMER:

I wasn't pooh-poohing it. I was asking you actually to elaborate on it.

LONG:

It doesn't fit in, it doesn't fit there. It doesn't fit in collections.

VELOSA:

Put it somewhere else because look, I put some things on the side.

LONG:

Yeah, I did, you know.

VELOSA:

I don’t see where you put it. Are these yours?

GRAY:

No, these are mine. This is down in kind of the analysis, how do we actually do something with it? What are we doing?

[Simultaneous comments]

VELOSA:

If it is very important. I would suggest you write it down, put it down, 'cause we need to address it at some point because architecture is fundamental.

GRAY:

What I conceptually see is, like we talked about before. We talked about the active and passive. And so I think the passive process is probably our first ask, kind of hierarchically, where you have the passive process where --

LONG:

Materials engineer. I understand. It’s like lots of complicated processes, right?

VELOSA:

-- you’re gather information, you know. So this is passive so --

[Simultaneous conversation]

LONG:

Unless you're just pushing atoms around or something.

VELOSA:

And when I say passive, it would be active monitoring.

LONG:

Yeah, yeah.

GRAY:

All of these, like data mining and different things like that. So what you’re trying to do is you’re trying to identify the big question and then on the next part, which is the active part, I think that feeds the active part, which says, you know, you’re trying to come out and say, you know, are we right, are we wrong, you know, and try to get the input from --

GOLDHAMMER:

We’re trying to elicit a response.

GRAY:

Elicit the response and then this comes back up to here and at some point, you know, you have to then say okay I have that input. Now whether -- you know, and again, up here, this is passive and this is -

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

VELOSA:

Yeah, you have two types of sonar, right? This is the active pulse that’s the--

GRAY:

Yeah, so this is passive ID and this is for technology, applications, etc.

BLOUNT:

I think that one thing is missing, a step ahead of that, which is you’ve got to generate a hypothesis before you start --

GRAY:

Right, right, hypothesis generation.

[Simultaneous comments]

GOLDHAMMER:

You’re at the head of the class.

LONG:

But you also need to have a team. I mean, to me that’s the problem.

GOLDHAMMER:

Yeah, so, Darrell, a part of what I was getting at is like what is actually happening there. Is it a team of people generating hypotheses?

LONG:

It’s as many as you can afford.

VELOSA:

Where are you going to put -- because I want to put this one with yours.

LONG:

And I don’t think – this isn’t necessarily linear.

GRAY:

No, no, no, it’s not linear.

BLOUNT:

I actually think this is passiveness.

[Simultaneous comments]

LONG:

So we’ve talked about this a couple of times, right?

GRAY:

Yeah, yeah.

LONG:

You have teams that are observing what’s happening in the world, right?

GRAY:

Arrows, arrows.

LONG:

And we didn't do this before. I mean, we’re being recorded but if -- look, if you look at the movies --

VELOSA:

We’re working on things that drive you nuts.

[Simultaneous conversation]

LONG:

-- the movies that are being produced right now, right? Al Gore goes off, rant, rant, rant, he’s Al Gore, and we get movies about the end of the world, you know, New York City being frozen after being flooded, right? In the 1950’s we’re scared of Sputnik. We get space alien movies, okay?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

So what you want is you want people observing the world, okay, the data’s coming in, and that feeds back into the kind of hypotheses that they’re creating. Hypotheses are generated. Other people that understand the technology or the social structure or whatever take those hypotheses along with data and say are these reasonable – what do we call them, Stewart? -- long bets to make. All right. You know, we start looking at climate things, for example, and we believe that this is going to cause starvation in Western China, which is going to cause migrations of people, which is going to cause social pressures, which is going to blah, blah, blah, blah.

GOLDHAMMER:

Yeah. No, I think that’s right. I think that’s right. So there’s a hypothesis engine that sits on top of this, which I think is right.

LONG:

Well, it’s not necessarily on top of it. These things are happening in parallel, right?

GOLDHAMMER:

Yes.

LONG:

And with feedback loops.

[Simultaneous comments]

VONOG:

beause usually this is [..?..] and this is arrows. Like this is a point and we have both at the same time --

GOLDHAMMER:

Arrows? We got arrows right there.

VONOG:

Oh, arrows? [..?..]

LONG:

So hypotheses inform data gathering, data gathering informs hypothesis formulization, okay? And then there’s an evaluation step which says this is a reasonable hypothesis, I’d like more data on this, or this is complete nonsense. Go away. The Chinese are not building a time machine.

GOLDHAMMER:

Right, at least that we know about.

[Simultaneous comments]

VONOG:

residual trends. Can we put trends and see where they come out

LONG:

And violence causes. They’re not building a time machine.

CULPEPPER:

So we’ve got the hypothesis engineer, you’ve got the passive and active listening, you’ve got --

GOLDHAMMER:

Guys, can you help us kind of sort of put some organization to this?

GRAY:

All right, so --

GOLDHAMMER:

These things are all, this is a system, right?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

So by active do you mean it’s okay if we tickle the system and see what happens?

GRAY:

Yeah, that’s what I’m saying.

LONG:

Right? We go poke the monkey with ….into Iraq?

GOLDHAMMER:

This is poking the system.

PAYNE:

Darrell actually is going to raise his hand to volunteer. [Chuckles]

GOLDHAMMER:

So take a look at some of these. Where do they fit? Do they fit in the passive or active part of the system?

CULPEPPER:

I’d put media tracking, passive, culture media, passive. I’d put , passive.

VONOG:

I think there should be like trend …., evaluation

GOLDHAMMER:

U.S. foreign media movie tracking, passive.

GRAY:

Passive.

PAYNE:

I’m sorry. Stan said something. What’d you say?

VONOG:

I said like maybe we should put like a trends, evaluation, like identification and evaluation. Like for example, when we’re designing products we’re thinking what trends are leading us–

[Simultaneous comments]

GOLDHAMMER:

Enthusiast tracking, passive?

GOLDHAMMER:

So that one’s active.

GOLDHAMMER:

Enthusiast tracking, passive?

CULPEPPER:

Yeah, passive.

VONOG:

And then you kind of build for the future, not for now, because it will be obsolete by the time you have this product ready.

GOLDHAMMER:

Precursor tracking?

CULPEPPER:

Still passive.

[Simultaneous comments]

GOLDHAMMER:

Story generation is definitely active, right? Or there. Okay.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

CULPEPPER:

Story generation, I like that. Competitions inside the military, prefer enlisted folks. Is that active? Or competitions?

[Simultaneous comments]

UNKNOWN:

[..?..] [..?..] board, media. That’s passive, I would say.

[Simultaneous conversation]

VONOG:

Well, I mean we are designing our product right now so if we would be designing for like today's [..?..]

GOLDHAMMER:

Irritators, that’s definitely active, right?

VONOG:

By the time we're done it will be obsolete.

PAYNE:

Right. Right.

VONOG:

So what we have to do is kind of hypothesize, see a trend [..?..]

GOLDHAMMER:

Yep. I think -- Where did Harry go? Harry, are irritators active or passive, in terms of sort of gathering -- Well are you pinging the system or are you [..?..]?

[Simultaneous conversations…]

VONOG:

It's when you have hypotheses then you look at trends and see if it's going to happen or when and how [..?..]

GRAY:

So here are these to add in somewhere.

GOLDHAMMER:

Team stewards system design, where do we put that? Maybe under hypothesis engine?

LONG:

Who is red?

GOLDHAMMER:

I don't know, it’s –

[Simultaneous comments]

GOLDHAMMER:

Team steward system design? You want that –

UNKNOWN:

Yeah, that would be hypothesis engine, yeah. And the same thing for that.

GOLDHAMMER:

Hypothesis evaluation. So these are precursors, application valuation, system architecture. Who did system architecture?

BLOUNT:

Darrell, was this you, system architecture?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

Yeah, yeah, that’s just a reminder that we have to come up with some sort of system architecture. Okay.

GOLDHAMMER:

Collaboration tools, people networks. Who did this one, collaboration tools and people networks?

UNKNOWN:

Who’s orange?

GOLDHAMMER:

Collaboration tools and people networks, who was that?

[Simultaneous comments]

VONOG:

Trends is different from hypothesis, trends is like what's happening.

UNKNOWN:

I thought we had that under --

UNKNOWN:

Somewhere in there I think?

PAYNE:

But no, he’s saying that that continues throughout the process so that you don’t – you reevaluate your hypothesis throughout the whole process. So that, you know, like say you’re building this for today but as you move along, a year or two years from now, the trends may have changed, you know, what your hypothesis may have been.

GRAY:

And that doesn't necessarily change, that doesn't change the system architecture. Yeah, it doesn't change the architecture. It’s just a matter that the output, the output comes back in here.

PAYNE:

Maybe this feeds into this.

GRAY:

Well and the fact that this actually feeds back and that’s the –

VONOG:

I was just saying like a startup, the only thing we’d do [..?..] future forecasting is kind we are trying to build a product and we want to build it so that it’s actual three years after. So we kind of think, "Oh, this trend is important for us so we think it’s going to happen," and like out of five define three and really bet on them also.

BLOUNT:

I think we’re still missing a step at the top. Before you can actually generate big questions you have to understand who the customer is and what drives their world.

GRAY:

Well, and that actually, it’s kind of interesting 'cause that’s going to be that big, that’s going to be that big loop down here. ‘Cause we need to start with who’s the customer but then at the end the customer is also going to give us feedback to tune the system.

BLOUNT:

Right. But for instance, if you’re the stakeholder of a hospital versus --So I think you have to set with kind of a definition of impact, you know, what rocks their world, positively and negatively.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

[Simultaneous comments]

VELOSA:

So this is a double-loop learning cycle, right?

BLOUNT:

Right, but how can you ask a question, right -- I mean --

VELOSA:

You have to know who your audience is so it’s just essentially go back to what’s important to them but --

CULPEPPER:

I think quick big picture questions I think is, there’s a lot you can put there, right, you know, by sector. I would almost think of it as like, you know, my sector is bitter that different issues, you know.

[Report of Ft. Hood shooting, continuing side conversation about terrorist attacks.]

[General conversation]

GRAY:

Well what, Darrell, what did you mean?

LONG:

Identify the customer. Identify the stakeholder.

LONG:

No, no, that’s not what I meant. What I meant by this is there’s another thing up here, identify the customer. But then there’s going to be things inside the “in here” that inform -- you know, you’re interested in energy, okay, so you define this whole thing energy, right?

GOLDHAMMER:

In the context of energy, yeah.

LONG:

But there’s stuff that’s going on that’s going to affect energy, right? You know, there’s social things, there’s climate change, there’s all this stuff, and this is going to inform what kind of hypotheses get generated, okay?

GOLDHAMMER:

Yeah.

LONG:

So, you know, if we’re worried about drought suddenly and massive migrations of people, right, then we’re going to generate different hypotheses than if we’re worried about, you know, the Northern Hemisphere becoming frozen, okay? There’s going to be different sets of hypotheses. So these are the big sort of large motivating questions, right, that sort of set the stage for things, right?

GOLDHAMMER:

Uh-huh.

LONG:

Like I said, 1950’s movies, all about space aliens 'cause we’re worried about Sputnik, okay? What are we really worried about, okay, on a large granularity and then what hypotheses can we make about what kind of technologies or uses are going to get popped out and we use the data to inform that and then it gets evaluated, whether this is nonsense or not, my favorite being the Chinese time machine.

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GRAY:

Here you go. Here you go, Al. Yeah, here you go though. But the thing is, it’s expert versus every man.

VELOSA:

Okay, yeah. Well actually every man and then another one for generalist 'cause you want, you don’t want that distinction in there.

BLOUNT:

But again, the customer could be, it could, these could be your customers.

[Simultaneous comments]

LONG:

I’m not sure how these, certainly how these affect, right? These are people that generate these. These are generators of experts saying, "Oh, gosh, you know, we’ve got to worry about climate change" and, you know, everyman saying oh…

GOLDHAMMER:

I’ve got a job, I need a job.

LONG:

Yeah.

GOLDHAMMER:

How come I don’t have a job?

LONG:

You took my job.

GOLDHAMMER:

Yeah.

VELOSA:

It’s just like thank you for the rf. I need money now for my solar project.

GOLDHAMMER:

You know, if we’re worried about jobs, right, this leads to outsourcing and, you know, all these other things.

GRAY:

Thanks for air-dropping the iPods in the Sahara Desert. Unfortunately, what do I do when the battery dies? [Laughter]

GOLDHAMMER:

What’s the connection between this sort of stakeholder audience, which is sort of -- that’s the connection between that and this part?

VELOSA:

Because they start the big picture questions.

GOLDHAMMER:

They’re asking the big picture questions.

GRAY:

Right. So I think –

BLOUNT:

So it’s the big question to hospital is different than to a city, than to the globe.

VELOSA:

Right.

GRAY:

But I kind of got the impression that we wanted to, we wanted to try to vet the big question.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

VELOSA:

Oh, I see what you’re saying.

GRAY:

With, with these, within these levels of expertise. We wanted to vet the big questions, okay, or get buy-in. I mean, you know, because I think the first thing you have to do is, is there – what is the big question and if you’re identifying the big question, what’s the big question for whom?

VELOSA:

And even iterate it.

BLOUNT:

Well, but if you’re going to vet it though, it’s almost, you almost have to start with the hypothesis before you vet.

GRAY:

That’s true.

BLOUNT:

So I think the question -- it feels to me like they belong up here or they belong --

GRAY:

Okay.

LONG:

Hold on, hold on, hold on.

LONG:

So for me this means --

GRAY:

Let me put them down here in the analysis.

BLOUNT:

Well, yeah.

LONG:

So for me this box says what am I worried about. What keeps me awake at night?

CULPEPPER:

Or I want to conquer the world.

LONG:

Or sure, yeah, world domination is always in favor [..?..]. [Laughter] So this keeps me awake at night either plotting or worrying, okay.

GRAY:

And the stakeholder gives you guidance as to where you’re going to be.

GOLDHAMMER:

So this is really setting boundaries, it sounds like.

LONG:

Right. So this is --

GOLDHAMMER:

What we care about, what we don’t care about.

LONG:

Well, not so much boundaries as a basis of what we’re thinking about.

CULPEPPER:

It’s a frame that you hang things on.

LONG:

Yeah. So you know, Stewart’s over there but climate would be a big thing here, right? And so everybody’s worried about climate so that’s going to cause the hypothesis generators to generate things that are

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

related or could not necessarily cause this but are caused by it or impacted or whatever.

GRAY:

EPA comes in and says this. NIEHS comes in and says this. DOD comes in and says this. But we still run it through the same process.

LONG:

But it could --

VELOSA:

Or Senator Acme says, "I need to get re-elected" or "My constituents need jobs --

[Simultaneous comments]

GRAY:

My state says this.

LONG:

Okay, but I think, and I think it can be broader than that, okay, not just the stakeholder says this, right? We’ve got to be smarter than them, right? They have a narrow view of what they want, right? We have, you know, during the Cold War what did we think about? Oh, the Russians, the Russians, the Russians, right? Couldn't think about anything else. We actually need to think about things though. Then the Chinese and the Russians at this point, you know, we need to think broader. So don’t let them say –okay, this is the thing for following the Chinese, right?

CULPEPPER:

I mean, that could be anybody, right? Like it could be a monopoly, it could be --

LONG:

It could be a monopoly, it could be anything like that but what’s the broad questions that you frame it? Here’s the technologies that are popping up, right? You’re seeing them passively, you’re just kind of noticing, watching the literature, the money flows, whatever. Here, over here you’re pinging the system, right? You poke the monkey in the cage and see how it reacts.

GOLDHAMMER:

Uh-huh. And then it comes down to a -- so then these – we generate a bunch of hypotheses based on that information collection and then it gets evaluated down here.

LONG:

Yeah. These are the science-fiction writers, these are the guys that have happy dreams and bad dreams, okay, and say that I’m worried about this happening. These are the people that can analytically look at things coldly and say, "The Chinese are not building a time machine, that violates causality. It’s okay, you don’t need to worry about that thing," okay? But the biologist, you know, monkeying around just manufacturing, maybe you need to worry about that one a little bit. Okay? That’s these guys. And then there’s a feedback loop, right, feedback loop coming back here and they’re saying to the hypothesis guys, "Don’t worry about that, okay? Time machine’s not going to happen, right?" There’s a feedback loop that says, "This is very, very interesting. What you said is an interesting hypothesis. I want more data," okay, and it feeds back here.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GOLDHAMMER:

Where is it? Show me that feedback loop again.

LONG:

This loop goes back here and here and here.

VELOSA:

But it goes all the way back here 'cause, you know, every so often you’ve got to check the stakeholders and make sure this is still relevant.

LONG:

Yeah, yeah.

BRAND:

So are the evaluated hypotheses the product?

LONG:

They could be.

CULPEPPER:

I kind of think it is.

LONG:

Maybe there’s a little bit more happening here.

CUKPEPPER:

Yeah, maybe there’s something on the tail end there.

BRAND:

What is that? What comes out of this is what these people want, presumably, or is it something else? Or is there a context builder down here?

LONG:

Yeah, I think there’s a context builder. There’s somebody that’s like this down here saying, "That’s very nice but we don’t really care about that," or "Oh, my gosh, that’s a big deal."

GOLDHAMMER:

So what prioritization --

LONG:

There’s a policy, there’s a policy kind of -- this is a technical evaluation, okay? There’s a policy evaluation down here.

GOLDHAMMER:

Let’s capture that.

GOLDHAMMER:

Is everyone in agreement with that, by the way?

[Simultaneous comments]

PAYNE:

The output is investments?

VELOSA:

What is the output? At the end of all this work?

GOLDHAMMER:

Darrell, would you say that again just to put a stake in the ground?

LONG:

So I’m saying this is more of a technical evaluation. Here there needs to be a policy evaluation step, okay?

GRAY:

Policy…?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

Policy, you know, decision-maker, you know, humanities major, lawyer kind of evaluation, right, those guys that actually control the money and don’t like the decimal places, okay? Is this going to affect the policy decisions?

BRAND:

So these are the storytellers that go ahead and talk to these guys?

GRAY:

I don’t, I don’t see that as linear. I see that as – I see this hypothesis evaluation as being multiple sources of evaluation.

[Simultaneous comments]

GRAY:

So you have a technology evaluation, you have a political evaluation, you have, you know --

PAYNE:

I would think you should.

LONG:

In my experience, these people are not particularly imaginative.

PAYNE:

That’s true.

LONG:

So for me – you guys can overrule me – these are the guys that are imaginative, okay? They’re coming up with, these guys are the guys that have nightmares and happy dreams that think of "Oh, my gosh, what could happen?" And I don’t want to constrain them. I want them to say, “What could I possibly do with all these wonderful technologies that are popping out?” Okay? If I put these policymakers and stakeholders in here then they mess things up

[Simultaneous comments]

PAYNE:

That’s why I thought they were down here because they’ve got to put –

LONG:

No. These guys don’t know anything about decimal places, okay?

PAYNE:

Well I know but they do know about how -- one thing they know is –

LONG:

Put them down here, okay?

PAYNE:

Okay.

LONG:

I want to put them down here because I’ve seen this many times where you take and you go and you scare the humanity majors, okay, and they go, “Oh, my gosh, could that happen?” And it’s complete technical nonsense! okay?

GOLDHAMMER:

So you’ve got a bunch of different -- this is an evaluative?

GRAY:

Right. So this is the eval- and I don’t like -- I mean, again, I go back to it’s not technical. Hypothesis evaluation is all of these things, scitech, gaming and crowd sourcing, social --

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

So these things are really glued down to the bottom of this, okay?

GRAY:

Okay.

LONG:

Okay, so technical’s the wrong word.

VELOSA:

Because then this has to have outputs that are then, you know, policies, priorities, whatever else.

GRAY:

Right, right. And that’s what I kind of threw down here, is these people, at this level we need to come up with a scoring system.

LONG:

So these are clever expert type people.

GRAY:

Yeah, yeah.

LONG:

Okay? So this is expert evaluation, all right?

GRAY:

And this brings in the non-experts so this brings in -- the generalists, the everyman is coming in in this piece. And then we have some sort of scoring system. I mean, when you throw a set of assumptions into a gaming system there are going to be a group of people that say, “The game’s wrong because this and this and this are screwed up” and they’re going to throw it out and dismiss it and maybe that’s what we need to take away from it is, we need to throw those out and dismiss it and go back.

LONG:

So you’re saying, for example, this financial person can say, “That’s never going to happen because this is going to cost a billion dollars,” right?

GRAY:

It’s going to cost a billion dollars or for that, like the guy on his own in Montana who has no connections to anything to bring his technology to light, it’s not going to happen.

LONG:

Yeah. He publishes it and then --

GRAY:

But somebody else does, that’s a different story.

LONG:

Yeah. Right. I’m a little worried about the everyman 'cause to me that seems it’s adoption issues, right? We’re not making iPods here.

VELOSA:

No, I get that but, you know, there’s a certain group think that elites have and I’m including us in the elites just 'cause we’re way overeducated.

LONG:

Yeah.

VELOSA:

Right? this room is just way overeducated –

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

Yeah. I’m happy to have these guys here but I’m just wondering exactly what their role is.

GRAY:

Well I mean, it needs to be -- ultimately what comes out of cloud sourcing and gaming is going to have to be evaluated and fed back into the system with some sort of rank score.

LONG:

Right.

GOLDHAMMER:

Right? So –

LONG:

‘Cause, you know, these guys --

PAYNE:

And they have to be in here or then again you’ll have bias in here too, if you don’t have these folks in there somewhere in the process.

GRAY:

Right. So this I see as coming out and these are the people that help to remove some of the bias that’s inherent up to here.

PAYNE:

Right.

LONG:

Right. But some people need – you know, I’m a technical guy, sorry, right, and --

VELOSA:

Right, but that’s the point of going to the favela. I mean, would you have thought of trying to shoot down that helicopter? I mean –

LONG:

Well yeah, actually, but --

GRAY:

Yeah, but that’s beside the point.

LONG:

But that’s beside the point. That -- by the way, for me, okay, these everymen and everything, they’re back up here in hypothesis generation, okay? They’re coming up with clever ideas. They’re not telling me whether it will work or not.

PAYNE:

Well. Sometimes it will because if there’s no political or social economic will to do something then that may lower it on the priority. It may not eliminate it because it’s still a possibility but it may lower on the priority on where you put your bets.

VELOSA:

But I think Darrell’s point is very valid. In fact we should have them here and there.

PAYNE:

Right, exactly. Right, they shouldn't come in. They should be involved somewhere in. They have to be involved.

[Simultaneous comments]

LONG:

Let’s do my Chinese time machine. Oh, crap, these guys are dreamers.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

VELOSA:

You know, you keep raising it and then you keep telling us it’s impossible.

 

[Laughter]

LONG:

No, I just took physics. Okay, look, so but these guys are dreamers, okay? They make up some, “Oh, my gosh, we see all of these things” – maybe the Chinese time machine, right, and these guys are going to come down here and the gamers are going to say, “Oh, yeah, time machines. Those are cool.” Okay, and they’re going to want to do this and these guys are going to -- the only people who are going to say no to the time machine are here.

VELOSA:

Well, Neil Stephenson proved that it worked. Ehhhh.

BRAND:

Two more questions.

GOLDHAMMER:

We’ll talk about Neil Stephenson later.

[Simultaneous comments]

GOLDHAMMER:

Two more questions, here we go.

BRAND:

Two more questions.

LONG:

So I think these guys are here, right, but then there’s – some judgment has to happen. But these guys are also here, you know, saying, “Well we've got all this cool stuff. What cool stuff can we do?”

GOLDHAMMER:

Okay. Questions, Stewart?

BRAND:

Two more questions.

[Simultaneous comments]

GOLDHAMMER:

Hold on, Guys.

BRAND:

Okay, there’s two categories that don’t exist here yet. One is the output. Maybe that’s where they were talking about. What is the output of this which is going to make the stakeholder audiences happy, and are we going to play the persistence gig, which is our specialty?

SAFFO:

In the spirit of that, I have an output for you.

VELOSA:

Oh, okay, that would be a better place for that.

SAFFO:

Pitch movie proposal to Dreamworks.

GRAY:

[Laughter] I love that.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BLOUNT:

[Chuckles] Nice, nice.

BRAND:

That’s fair and then actually make the trailer and show it to the audiences.

BLOUNT:

I love that idea. Seriously, I love that idea.

GOLDHAMMER:

What is – other than the movie proposal to Dreamworks, what other outputs are there? There might be multiple outputs.

VELOSA:

I think there’s new questions.

GOLDHAMMER:

New questions as part of the feedback.

[Simultaneous comments]

BRAND:

The thing is, there’s some entity that decides what the outputs are and this is the judgment.

CULPEPPER:

I think one of the outputs is a long bet.

PAYNE:

Right, is the forecast.

BRAND:

That’s fair. I like that.

CULPEPPER:

The long bet, the collective entity that says, “This is the long bet”.

UNKNOWN:

Long bets.

SAFFO:

And for those of you who haven't seen long bets, the longest bet on the site is a bet between Danny Hillus and Nathan Miravel over whether the universe will stop expanding or not. And Nathan claims that he’s already won the bet and Danny just doesn't understand the question.

BRAND:

Nathan is ahead right now but that could change.

BLOUNT:

But in all seriousness, to Paul’s note on Dreamworks is, you know, one piece of feedback you could create is to create a movie short, put it out on the Net, or create a two-day game ala, you know, some of the simulations that are out there and get people to live it.

PAYNE:

Yep.

BRAND:

Actually everybody can make their own videos now. Whoever lives down here in this part of the company just make a video and that’s what you send over to these guys, probably not to the world 'cause some of them will be quite threatening and –

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BLOUNT:

Right, but if I think of the Army -- what’s the Army game that was created online?

[Simultaneous comments]

LONG:

America’s Army.

BLOUNT:

America’s Army, yeah.

BRAND:

That’s another thing you can do, a game.

LONG:

Much beloved by the Germans.

BLOUNT:

You could create a lot of different scenarios within a framework.

BRAND:

So how does persistence play in all this? Just do it again and again or what?

PAYNE:

That goes with the new questions.

LONG:

This is coming in all the time, the unblinking eye.

VELOSA:

The eye in the sky, right?

LONG:

Unblinking eye.

VELOSA:

Oh, so that’s your “always data coming in?”

BRAND:

Always data coming in.

[Simultaneous comments]

VELOSA:

Right. So there's data coming in, what are you doing with your hypothesis over time.

LONG:

These are hypothesis generators.

BRAND:

Are they looking at the output as well as [..?..]

VELOSA:

Well, you do get new questions. But you do get new questions.

LONG:

Exactly. We need at some point to get a loop all the way back up to [..?..] at some point, somewhere.

GOLDHAMMER:

There you go.

GRAY:

So I talked to, I was talking to Harry earlier and that is one of the, one of the – it was, Techcast, was the idea of coming in –

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BRAND:

That’s the reporting to them. We want feedback that goes through these guys and all of that output there [..?..] constantly looking at.

VELOSA:

Well then go both to stakeholders and then to --

LONG:

It needs to go both, not through.

VELOSA:

Oh, I see what you’re saying.

GRAY:

Okay, so it has to fork.

LONG:

If it goes through these guys, and no offense to the policymakers, but they’re not the most imaginative people in the world.

[Simultaneous comments]

VELOSA:

Actually, give me two arrows.

GOLDHAMMER:

Oh, give you two arrows too.

VELOSA:

Yeah, just give me two 'cause then we’ll point. This is a fork. It’s not .

GOLDHAMMER:

Arrows are right here.

CULPEPPER:

So to the question on what the output is, shouldn't we also have some sort of like -- Out of this whole process, what was our odds on after year one, year two, year three, year five, year ten, right? Basically what you do with long bets but formalizing that and say you know what we’d really like to do is we’d like to be on mark, you know, out of this engine 80% of the time or 70% of the time or whatever the number is, right? ‘Cause right now there is no odds on. I mean, that’s kind of my takeaway from all this. The whole reason this exercise is occurring is 'cause people are not happy with the outputs they’re getting right now, right, or they’re not getting outputs at all, you know, or they’re so wildly off the mark that they need better

[Simultaneous comments]

BRAND:

So what’s the nature of the effective forethought

GOLDHAMMER:

Well that’s -- So here’s –

BRAND:

Good policy, right?

GRAY:

This goes back to --

LONG:

Yeah, yes.

BRAND:

Leading the world peace. I have it down there [..?..].

UNKNOWN:

Yeah, yeah, all right.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GRAY

This goes back to something I was trying to get back to and that is something quantifiable and that is where I brought up Techcast, was what -- The presentation we got for Techcast was you get all these things, you have the experts rank it and then what they delivered was consensus of the experts. I don’t like that. One, I don’t want to trust everything to the expert, which we’ve built into the system we’re not going to do. Two, I don’t like throwing out the outliers. But if we generate a ranking analysis and we report both the consensus and the outliers with the scores, then we can go back in each iteration and say how successful was the consensus in prediction and how was the outlier in prediction.

CULPEPPER:

Right, right.

BRAND:

Hovering right about here, the brilliant fucking leader, and hovering right about here the great storyteller who's often a different person.

[Simultaneous comments]

GOLDHAMMER:

I actually think there is a narrative engine or some narrative component to this down here. I think someone has to generate that stuff.

GRAY:

I think that’s, I think that’s here. I think that’s where the narrative comes from is that now you have all of these, you have all of this as outputs but you need somebody who’s going to take this and this is where that loop then comes back.

GOLDHAMMER:

So this loop here?

GRAY:

Yeah, so the narrative --

LONG:

Be really careful with that loop, okay? Because we forked back here. I think --

GRAY:

Okay, so maybe the narrative –

LONG:

These guys get different than these guys.

GRAY:

Yes, the hypothesis engine, these guys get the raw outputs –

LONG:

Yes, this goes here.

GOLDHAMMER:

These are raw outputs?

LONG:

Yeah.

GOLDHAMMER:

So this is the output so this goes –

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

VELOSA:

So wait. Do you want me to write “raw” for the hypothesis?

GRAY:

Well you can say “raw”, yeah.

BRAND:

Yeah, you’re right. Raw goes to the hypothesis and story goes --

LONG:

Right, and then, and the narrative output goes to the stakeholder.

GOLDHAMMER:

All right. So this goes up to stakeholder?

GRAY:

Yeah, this goes up to stakeholder, yeah.

LONG:

So this, this is – this is after the decimal places are removed.

GOLDHAMMER:

Yes, yes, thank you.

BRAND:

And if it all works, then we get the product.

GOLDHAMMER:

Yeah.

CULPEPPER:

Or should we just write “the increasing DOD budget” on there?

BRAND:

No, wrong.

CULPEPPER:

I guess that’s on the record too.

LONG:

This is where you put pictures of polar bears.

GOLDHAMMER:

That’s right.

LONG:

The lonely polar bear sitting there, right? This is – the real data goes back here and the lonely polar bear picture goes here.

BRAND:

No, the happy large liver.

GOLDHAMMER:

I’m going to kind of, I’m going to convene this meeting back together and I think what we’re going to have to do is it’s going to be like a little bit of a movable feast. I think we’re going to just go from table to table to table and see what other people have done.

LONG:

Okay. Let’s see. There’s one more thing here. Where’d Harry go?

BLOUNT:

I’m here. I’m hiding.

LONG:

Okay, so I think Harry, you or Danny were saying this, right, about what comes out here in the measureableness of this and what I don’t want to do is say these guys were wrong, I’m throwing it out. They might just have been wrong on the timing.

BLOUNT:

Yeah.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

Right? You know, you might predict flying cars by 1970, okay? It didn't happen.

BLOUNT:

And we got ripped off on that one.

UNKNOWN:

We got what we’ll never have flying cars.

BLOUNT:

I mean, the real life example is the Twin Towers were built to withstand an airplane. We didn't continue to ask the question from 1960 in the design principle to monitor certain conditions.

LONG:

Right. So, you know, a prediction, for lack of a better word, or forecasting come through this, okay, and the idea’s right. It’s just the timing’s wrong. So if we say “I predict, you know, by 1970, you know, something”, right, we’re not Criswell, right?

BLOUNT:

Well, but if you frame your narrative right, here’s the thesis, here’s kind of what has to happen in the story line and through what has to happen in the story line you get your enablers and inhibitors out of it that then create the tracking tools to go up in there.

LONG:

Right, right. So this guy talking to the senators and the Congress critters doesn't need to bring everything forward, okay?

BLOUNT:

No, no. But I –

LONG:

It needs to bring forward the compelling things.

BLOUNT:

But the narrative engine has to, what has to be the key elements of the narrative has to be, you know, what has to happen –

LONG:

Yeah, right.

GRAY:

It has to have a recommendation of how to use the information.

LONG:

Right? I mean, what I don’t want is something to come here that it’s possible but low probability and this guy spits out a boogeyman.

GRAY:

Right.

PAYNE:

But – and it depends. If it’s low probability, it has to have high impact if it’s going to be important to us.

CULPEPPER:

Right. Yeah. But –

LONG:

But you know, it’s more than just impact, right? There’s a cost associated with things, right? I mean, I fly a lot and it annoys the hell out of me that I have to take off my shoes and do all of this other stuff. I fly in Europe, I don’t have to take off my shoes, you know. And realistically a lot of this is theater because somebody goes, “Oh, my gosh, if another airplane crashes into something it’s the end of the world.” It’s not, you know. Just

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

people need to be able to do the arithmetic on this. So I don't want the bogeyman to pop

[Simultaneous comments]

LONG:

You know, the most important thing that ever happened for airline safety is they made the door to the cockpit secure. Everything else is theater.

PAYNE:

Which, oh, by the way, they were told to do years ago but they didn't want to do it because it was going to cost $10 more per plane or something.

LONG:

Yeah. The rest of this is theater.

GRAY:

This really needs to include a cost/benefit analysis.

BLOUNT:

Which is what you do with your portfolio.

GRAY:

Yeah, but I think it definitely is a cost/benefit analysis. This is a weak signal but I'm [..?..]

PAYNE:

Yeah, you’re right, you’re right. But hopefully, if you’ve done this process right, by the time you get down here you have something that’s beyond theater. It’s actual probability.

[Simultaneous conversations]

GRAY:

See, we brought this up. [Chuckles]

BLOUNT:

We got it, man.

GRAY:

We’ve got to have it in there.

BLOUNT:

Yeah.

GRAY:

Because that’s the, that’s, this is -- I think within this what we’re able to do is we’re able to take racial, socioeconomic, you know, educational bias out in this step.

BLOUNT:

That’s why the openness part is so critical.

GRAY:

We still haven't addressed how we’re going to make it beneficial for some of these players --

GOLDHAMMER:

All right, we’re going to do the first -- actually, Darrell, would you mind, could you walk people through what we’ve done?

LONG:

As long as I get a little help from a couple other people, right?

GOLDHAMMER:

Absolutely. That’d be great.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BRAND:

What kind of things -- our deal was persistence, which means feedback, and what part of the output should go back into the hypothesis engine which drives the whole thing, part through the narrative that goes back to the audience.

SCHWARTZ:

I’ve looked at all three. Everybody took a different approach. That’s perfect, exactly what we were looking to happen. So when we meet tomorrow in the committee we actually have some very good options.

Feedback to workshop participants in Appendix D.


Team Activity: Identifying the Human and Technical Requirements

GOLDHAMMER:

All right, where’s our team?

PAYNE:

They quit. [Chuckles] They’re resting on their laurels.

GOLDHAMMER:

Oh, they are. Look at that.

GRAY:

They’re contract holdouts. They’ll be here eventually.

GOLDHAMMER:

Yeah, they’re waiting for their bonus. All right so –

PAYNE:

Move the coffee and doughnuts over this way.

[Side comments continue]

GOLDHAMMER:

So I guess two choices. The first question is there energy within the group to try to revise anything that we’ve done on the back end? There was a suggestion from, I think it was Stewart, actually, he made --

BLOUNT:

He made it but I don’t think we need to do that. That can be done by the committee tomorrow.

GOLDHAMMER:

I don't know whether we need to do that. Is there any energy to sort of, to flesh this out a little bit back here, like what the actual outputs are, the reports or something like that?

SAFFO:

I think Stewart’s suggestion is great but I think it’s done --

BRAND:

Grab their four things, plop them on there, we’re done.

GRAY:

Yeah, I mean, there you go.

SAFFO:

This could be done digital.

VONOG

Maybe also run a well known thing like conventional cell phone release to see like hypothetically if it could sort of come through this process or something like that.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GOLDHAMMER:

Okay.

GRAY:

So you’re saying come up with –

VONOG:

A story which is well known like, I don't know, Google, Apple, cell phone, whatever.

GRAY:

Yeah. So maybe take three sample inputs. Take a technology input, take an application input and take maybe a, you know, a market combination of what –

VONOG:

Yeah, but someone has to know the story, what was like the environment before that.

GRAY:

Right, right.

PAYNE:

A business case where you strip all the names.

GRAY:

Yeah, and feed it through and then see how well it comes, well how well it scores through our system.

PAYNE:

Right.

GOLDHAMMER:

Let’s do that and let’s do, actually let’s multitask. I know they say anyone over 30 can’t do that. Let’s multitask. Let’s actually run a technology through it, but then I’d love to also get a sense of who’s doing what, whether there’s – let’s see if we can figure out what are the technologies, what are the people, what are the partnerships, what actually has to happen. And I’ll try to record some of that as we walk through it, okay?

GRAY:

Okay.

GOLDHAMMER:

You want to pick -- The one thing that, you know, I don't know if, Ken, this would be helpful for you, but I was wondering if we should walk IED, like let’s pretend it’s, you know, 2000 or something like that, pre-IED world. I mean, that has to be the most disruptive technology –

VONOG:

And maybe you tell like a little bit of the story–

[Simultaneous comments]

LONG:

It’s also technology from, you know, for fighting in the 1970’s when people were slogging around Vietnam and we didn't solve it then either.

PAYNE:

I don't know if IED’s a good test case.

LONG:

That wasn’t a surprise.

GOLDHAMMER:

Why don’t you pick one then, pick a test case.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

But I think it’s important. IEDs were not a surprise. They weren't.

PAYNE:

Not even close to a surprise.

PAYNE:

And see the reason, and one of the reasons they were disruptive too is that Humvees were built to be supportive vehicles, not in the midst of war. So it wasn’t like, "Hey, you didn't build what you needed." It’s, you know, everything else changed too.

GOLDHAMMER:

That was the soft side, yeah.

PAYNE:

Right, and it took advantage of that.

LONG:

We came to fight a different war -- than we ended up fighting.

PAYNE:

Right. I’m trying to think of something [..?..].

LONG:

Okay, so here’s one for you. I’m doing timekeeping down to the "Oh, I don't know," second level, okay? And people, you know, your observers see that I’m doing that kind of [mike noise], okay? [mike noise]

PAYNE:

That’s….

LONG:

Okay? And then somebody observes that we can launch satellites, okay? Isn’t that clever, okay? And other people realize that there’s digital communication with, you know, coding and so forth.

VONOG:

That’s hypothetical. I was thinking more like you know the story and you’re not going to see when –

[Simultaneous comments]

LONG:

I know the story. Oh, you don’t recognize the story yet.

VONOG:

Okay, okay.

PAYNE:

You will.

LONG:

Okay? And this guy just read the book Longitude, okay?

VONOG:

Uh-huh. And when did it happen, like fento second? Did it happen before –

LONG:

And maybe it’s not fento second. Maybe even off by a couple orders of magnitude, it doesn't matter. I’d have to look at what the seasoned clocks are right now.

VONOG:

And so like the high level story behind the was that they invented this tiny fento second, whatever.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

Right, so atomic clocks, okay, atomic clocks are the result of 30 years of research and timekeeping.

VONOG:

And when was that? Like can you tell who –

LONG:

Starting in – gosh, help me out here – sixties, fifties? Fifties or sixties. It’s a long-term story, until we got to the point where you could put these things on a satellite.

VONOG:

Uh-huh. So they were small enough or…?

LONG:

They’re small enough now that they can go on –

VONOG:

What were the metrics or something with this?

LONG:

Well they used to be, they used to fill this room.

VONOG:

I see.

LONG:

Now they can put them on a satellite. Actually now you can buy them and you can put them in your routers, okay? And DARPA has them now that goes on a chip, Butnet.

VONOG:

Yeah, in watches, they have, the Citizens, Citizen –

LONG:

No, no, no, no. Those aren’t, those are not atomic clocks. Those are just things that get the signals from atomic clocks. Okay, so we have atomic clocks that are now miniaturized. We have space launch and somebody read the book Longitude, right, which actually is later, after these things really came out. But -- you understand the Longitude story?

VONOG:

And what’s the longitude…?

LONG

Longitude story is if you want to know your position on the Earth, and the British had a competition for this, to go north to south you look at the stars. East to west, you’re screwed, okay? So what you would do? So what you would do is you would see where I -- I’m in jolly old England right now and --

SAFFO

Screwed, you’re just screwed if you’re on a boat.

LONG:

Well I think you’re screwed if --

SAFFO:

You can use lunar distances if you’re on land [..?..]

LONG:

Yeah, as long as you just keep track of where you are the whole time.

SAFFO:

No, lunar distances from –

LONG:

What can you do?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

SAFFO:

-- as long as you have a table.

LONG:

As long as you have a table. But you have to know how far you --

SAFFO:

You’d have to be on land.

LONG:

And you’d have to know how far you went.

SAFFO:

You can use a stable platform.

LONG:

Yeah. You have to know how far you went, right? So the story is that there was a competition and the British government were giving out a thousand pounds or something if somebody could keep time accurate to, you know, a second per week or something.

VONOG:

So we would detect those kind of competitions through the system?

LONG:

Right. So you detect competitions like this. So that’s the story but that’s a long time ago. And what happened was this guy developed a clock and he would say what time is it in England and I’m in England, okay, fine. And you get in your boat and you go and you say, "Well it’s noon now and in England it’s 4:00 p.m. so I must be here," okay? So time helps you determine position, okay? So these guys dream up things, satellites and clocks and things like that and you get GPS popping out at the end ultimately.

PAYNE:

So what were the active processes claimed?

LONG:

I don't know if it – I don't know. Maybe --

PAYNE:

Engaged in academia?

LONG:

I mean, this is something, I’m actually not that clear on what active is. I think perturbing the system.

GRAY:

Yeah. This is where we were determining -- So actually the active system here is historically the competition -- The competition that someone read about historically was there but it was an active competition years ago. There’s no reason you couldn't do it now.

LONG:

Yeah, and you could actually start a competition if you wanted to see what you wanted to, you know, you wanted --

SAFFO:

Well you can summarize active competition with Arthur Clark’s famous quote, “The best way to predict the future is to invent it.” That’s the most extreme case.

GRAY:

There you go.

LONG:

All right, and these guys, they get this and they say well that’ll work or that won’t work or whatever and it comes out here.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GOLDHAMMER:

So the hypothesis, but just to be clear, so the hypothesis engine is actually saying well if you actually – if we saw someone playing around with, you know, highly accurate time, the ability to tell time, that could mean this, it could mean this, you could do it this way, you could do it that way.

LONG:

It could mean this, right?

VONOG:

Or maybe there is a big question first of time and then these guys [..?..] –

[Simultaneous comments]

GRAY:

The hypothesis engine of the time would be I read this book about this competition. I know that there’s this technology now that’s a very, very accurate clock. The question is –

LONG:

What could I do with it?

GRAY:

-- with today’s technology could I go back and do what they were trying to do 300 years ago, and if so –

LONG:

Actually end up doing it completely differently.

GRAY:

Doing it differently but could I do it? And the question is how would I do it and that comes down to here where you say okay, I have this hypothesis and it comes down to here and it’s evaluating the technology and the hypothesis together and saying will it fly.

VONOG:

And what happens next?

LONG:

I didn't make these triangles.

PAYNE:

Well but part of this is that where the –

LONG:

Technically these guys will evaluate is this going to work technically, right?

VONOG:

Oh, so these guys work here –

LONG:

And these guys are in here.

GRAY:

Yeah, yeah, these guys are part of this.

LONG:

These guys say, "Oh, is there a business case for this?" The answer of course is no but the government pays for it anyway, right? And then we find out later that it’s a multibillion dollar industry, but…

GRAY:

Well and then, you know, social, political sci-fi.

LONG:

These guys don’t have to agree.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BLOUNT:

I think one of the questions I have here is how much of this structure here in terms of the filters is internal versus external.

GRAY:

That’s a good question.

BLOUNT:

Because if it’s all internal you’re going to have, ultimately you’re going to have a structural bias of some shape or form and –

LONG:

I think it depends, right? Some of these things can be internal, some of --okay. Let me just speak about this one, okay? The federal government right now, the state of affairs and their ability to do science and technology evaluation is abysmal. You know, all the -- there’s no –

VONOG:

It’s bad or good?

[chuckles]

 

LONG:

It’s bad. There’s no path to stars or flag rank for an engineering PhD in the military. There used to be. There’s not anymore. Intelligence Agency have gutted this stuff so there’s a –

BLOUNT:

Complete inverse of China then.

BLOUNT:

Complete inverse of China.

LONG:

Complete inverse of China and frankly, it needs to get changed but we’re run by lawyers so who knows. So this is going to have to have some other inputs like the national labs or places where you have contract scientists and things like that doing this. But you can have -- so I think the government needs to improve their science and technology internally and they’re going to look at stuff and they’re going to see things and they’re going to say, “Huh, we don’t really quite understand this. Let’s go out and get some real external people to look at this.” And I think that’s reasonable and healthy. And probably financial is the same way, right? They really don’t have any great financial people, otherwise they’d be on Wall Street making big money.

PAYNE:

That’s not necessarily true. I think like these, on an S&T and the gaming and cloud sourcing, are both and inside and outside the government.

GRAY:

I think so too.

LONG:

Yeah, inside and outside, yes.

PAYNE:

But these, you don’t -- the financial part –

[Simultaneous comments]

GOLDHAMMER:

So is this Treasury and VCs or something like that?

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GRAY:

Oh, hold on. Let me -- Let’s do this, let’s do this.

PAYNE:

That’s probably not Treasury. ‘Cause every organization has their financial folks to help make those decisions.

GRAY:

Let’s put this over here.

LONG:

Yeah, good. Yes.

PAYNE:

And so -- I don't know how well we do that but we’ve got those folks.

SAFFO:

Post-docs.

LONG:

There you go. The disgruntled post-docs.

GOLDHAMMER:

Who are these people? Just put –

PAYNE:

They’re within each organization. Everybody’s got their financial folks that tell you whether or not you can do it or really what they tell you is here’s your tradeoff. If you want to do this, you can’t do this.

LONG:

So I don’t mean that kind of financial, right? I mean, here I mean like the VC kind of financial. Does this – is there a business case or financial pressure that would cause this to happen?

GRAY:

Cause it to happen or cause it –

LONG:

Or cause it not to happen.

GRAY:

Not to happen.

PAYNE:

But there’s some things, you know, you work -- Look at these two ways. One, if you’re concerned more that somebody else is doing this before you are, then your concern – then you say, "Hey, can this be done and what it's going to cost?" The other thing is can we do it first. You know, hey, there’s these two things together. Can we put this together and develop this system before somebody else?

GRAY:

But what the venture capitalists are going to help decide here is --

LONG:

This is a "can," this is a "will."

GRAY:

-- is this a privately funded venture or is this a government funded venture? Something might come through and the VC guys are saying –

PAYNE:

Right. That’s assuming somebody else is doing it before us.

GRAY:

-- I can’t touch it. Right, right. I can’t touch it.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

PAYNE:

That’s assuming somebody else is doing it. If we’re ahead of the problem or we think we’re ahead of the problem, we’re looking at can we do it and how do we do it and finance [..?..].

GRAY:

Well and that’s the question. And the VC guys are going to look at it and say, "Is this something that the private sector’s going to want to bid on, yes or no?" Even if we want to do it first, is the private sector going to want to bid on this or is this something that’s only going to fly if the government is there?

LONG:

So Harry, would you have put financial backing on GPS twenty years ago?

BLOUNT:

I have no idea, really have no idea.

LONG:

Probably – I would hazard they would not.

PAYNE:

Yeah, because the timeline on getting your return on investment is too long.

SAFFO:

Well but this is just a matter of granularity, it’s the scale problem when you’re deciding to invest.

LONG:

No, well this is a question of who’s going to do it, right?

PAYNE:

And the other thing too is that, you know –

LONG:

The government had a compelling need for this, right? It wanted the guys in the Army to know where the hell they are, okay? They didn't care about people driving around in their cars or me and my iPhone navigating the streets.

PAYNE:

That’s when Harry’s folks come in.

LONG:

That’s when they come in.

VELOSA:

No, no. But the semiconductor firms and some of the electronics firms were in there because one, they had products that were relevant and they had contracts.

LONG:

What is this?

VELOSA:

The semiconductor and electronic firms.

LONG:

In 1980’s?

VELOSA:

They were starting to work on some of the parts for it. They didn't know GPS but they had some of the parts for it.

BLOUNT:

I think we’re asking the wrong question.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

I don’t know which part you’re talking about.

SAFFO:

I wouldn't push too hard on the GPS example. It’s a nice one at a general level but it’s not –

BLOUNT:

I think you’re getting to the question of asking the right question at the wrong time. So if we’re designing this correctly, is GPS going to be feasible, is a disruptive? The financial guys may say, "Right now it’s not feasible but here are the signposts for feasibility." That in and of itself becomes a persistent question that continues to echo through and then when you hit, boom, a price point, the semiconductor guys start making investments, what have you. So I think these guys are here to say --

GOLDHAMMER:

So who are these guys? Who’s doing that?

BLOUNT:

I think in the social/political realm is if it is something like the National Highway System, you know, if one of the enablers is the National Highway System, that has to take some kind of political decision so that becomes an enabler in terms of length. That’s a big enough project.

PAYNE:

And like I said, those folks are kind of fairly senior folks but then whatever organization’s asking the question, you know, it wasn't -- In that case it was like Department of Defense was asking that question so then you’d have some of the folks probably in DDR&E that were involved in that and just OSD, Office of Secretary of Defense.

VELOSA:

But you’ve also got, you know, X congressman with his pet company and his territory is going and saying why are you buying?

PAYNE:

Yeah, but we can’t put them in the process.

VELOSA:

But they are in the process though.

PAYNE:

They’re wildcards though, they’re wildcards, 'cause you don’t know. There’s no logic behind –

UNKNOWN:

They’re external. Yeah.

[simultaneous comments]

LONG:

You wouldn't go and get X congressman and put him in your system, right? They’re external actors.

VELOSA:

They’re already in the system.

LONG:

Not in this system if we’re building a system.

[Simultaneous comments]

GOLDHAMMER:

Yes, yes.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

BLOUNT:

China’s going to need to consume massive quantities of resources over the next 20 years to support their economy. What are the key things as they go out and try to secure those resources to support their country long term, what are the social and political considerations that are going to enable the forks in the path?

GOLDHAMMER:

But who’s doing -- A question I’m asking, which is who’s doing that evaluation? Is that just inside the government or are these academics doing it?

[Simultaneous comments]

LONG:

No.

VONOG:

Maybe it's like intelligence

LONG:

I think you need to reach out, you need to be able to reach out to like external academics as well, right?

GOLDHAMMER:

Uh-huh. Yep.

LONG:

So for – whatever the experts are.

PAYNE:

National Academies.

LONG:

Sometimes but, you know, you need to be able to -- look, if you’re in the government and you see something going on and you have a concern, you need to be able to reach to somebody. National Academies has a long cycle time, okay? You should have your Rolodex, call up your favorite professor of, you know, minerals politics at a university and say, “Western Australia is being sold to the Chinese. What’s the impact here?” I mean, literally, that’s the – Australian economic boom is the fact they’re digging up all of Western Australia and shipping it to China.

GOLDHAMMER:

Gaming and -- I’m just going to keep you guys focused here. Gaming and cloud sourcing. In this product, who’s doing this?

PAYNE:

That’s internal and external because as far as Department of Defense is concerned, we do our own game in a lot of things, but we also use external organizations to do that as well.

GOLDHAMMER:

Contractors, yeah.

GOLDHAMMER:

So contractors?

GRAY:

So you can use, yeah, so DOD, you’ve got contractors.

GOLDHAMMER:

Okay? So this whole thing, I mean, the principle here is you actually have people inside and outside, across all the evaluation who are doing it. Is there any entity that is sort of managing this process that’s, you know,

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

sort of figuring out what questions go to what people, what answers are good, what answers are bad?

VELOSA:

I think it would be the team that – there has to be a team.

GOLDHAMMER:

Okay.

GRAY:

There has to be some sort of standing group that -- Because --

GOLDHAMMER:

It’s an evaluation group.

GRAY:

Yeah, that’s an evaluation group and they don’t necessarily do the evaluation. They coordinate the evaluation.

VELOSA:

But it's the stewards. They [..?..] the system design stewards.

GRAY:

They don't necessarily do the evaluation, they coordinate the evaluation.

BLOUNT:

Yeah, that’s critical.

GRAY:

They’re not, you know -- so the nice thing there is these can be information managers but not technical people.

BLOUNT:

Actually I would argue this is the National Academies’ role right here. A stakeholder comes to the Academy with a hypothesis, they bring together --

VELOSA:

Oh, like a TIGER committee for this?

BLOUNT:

Well or this committee or any other committee. They come with a hypothesis and the role of National Academies is bring a bunch of people together --

LONG:

Hopefully there’s way too much traffic going through there.

[Simultaneous comments]

VELOSA:

There has to be a standing group though. It can’t be just to come together --

GOLDHAMMER:

But this seems like a small group of people to me, you know.

GRAY:

Yeah, it does.

LONG:

What I’m saying is for every agency that’s interested in doing this kind of thing there’s a large number of hypotheses flowing through here being evaluated and meeting quarterly at the TIGER meeting isn’t going to do this for you.

PAYNE:

And, you know, one of the fundamental flaws within DOD is that –

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

I think the TIGER can oversee some big process like this.

PAYNE:

-- the science and technology folks, the senior scientists and technologists, they don’t have a, you know, they get together but then they have coffee and they just talk hypothetically, have really no power, almost no power to get anything done. And it’s actually the Acquisition Committee that actually has the power because they have the money. The money goes to them, you know. And even, they even control the DARPA money, you know, and so it’s, what happens is, and part of the hypothesis evaluation, the hypothesis engine is they gather – and we’re talking DOD – is they gather requirements, they gather, from the combat and command, which is like SOUTHCOM and CENTCOM, all that stuff, and they gather all the things from the intelligence community within the Department of Defense as far as, you know, concerns. And you need to have, where that comes together with science and technology has a role in it and right now it doesn't, it really doesn't. I mean, they give it lip service. They go, “Oh, this person’s the Science and Technology Advisor,” you know, and if they can get in and say, "Hi" and like, “Oh, yeah, what have you been doing?”, “Oh, we’re working on such and such, such and such, you know, and here’s some decimal points” and then, you know, that political appointee’s like, “Hey, man, nice talking to you. I’ll see you next month.” And that’s really the problem is, from our perspective, is getting those folks in with the folks who actually write the checks. Because everything that you buy today in the Department of Defense you’re going to take away from somebody else. That’s how it is.

GOLDHAMMER:

Yep, zero sum game. Yes.

PAYNE:

There’s a zero sum game issue. And I used to go to the guys who make decisions in the Department and say, “Hey, we need more money for this program.” And they’ll say, “Okay, where are you going to get it from?” I say, “I came to you. You know where it is?”

GRAY:

I see this, I see this, the management of hypothesis evaluation, the hypothesis engine, I see this as being an entity, whether it’s a contracting firm, whether it’s some sort of internal organization within –

PAYNE:

You don’t want it to be a contracting firm.

GOLDHAMMER:

Well no, I’m just saying but, you know, inside the, you know, whether it’s a new sub-piece of the TIGER committee or something like that.

PAYNE:

Well here’s the answer for today. Today this will come from the Acquisition Committee, okay? What needs to be added in here –

LONG:

Well the money will but the staff, you staff over there.

PAYNE:

The money and the decisions and some of the hypothesis generation will actually come in there. But, you know, maybe DARPA. DARPA has some say, but – as far as where they want to go. But there needs to be

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

more emphasis on the science and technology, the scientists and technologists, within the Department of Defense, have a say in that.

CULPEPPER:

Let me say something though. I think one of the -- when I look at this I think of Hollywood and the whole market of Hollywood, right? And these things happen all the time in Hollywood and there’s an entire ecosystem that’s built up around it and if there’s a way that you could replicate that ecosystem, you know, be it DOD, be it Department of Transportation, be it the private sector, finance, it doesn't matter, where you can basically have the -- because the enablers are the active and passive data feeds, right, and then a vested set of interested bodies who are generating the hypotheses, right?

GRAY:

But see -- I didn't mean to cut you off earlier, Ken, but what I’m saying is we need to think of this as coming out of our report. We need to think of this as DOD, you know, and the Acquisition. That’s great. That’s one potential customer. The Federal Reserve, you know, somebody else could be another customer as well.

PAYNE:

But DOD is paying for this.

GRAY:

Well I understand that.

PAYNE:

And the DOD wants to know -- because the Federal Reserve is different than DOD.

GRAY:

Well, I understand that but the architecture of the system –

VELOSA:

[..?..] DOD if it has multiple partners on this.

GRAY:

-- the architecture of the system is --

GOLDHAMMER:

I want to interject, I want to interject because I want to just kind of focus in a more practical matter. There are lots of things that we’re saying that we’re doing here and the conversation about who ultimately is going to pay for it or what that might mean is important but actually what would be very helpful right now is to figure out what exactly is the staff that does these things.

LONG:

Right. So I don’t think this is going to be staffed with acquisition officers, okay? It’s the wrong kind of person, okay

PAYNE:

I agree with you.

LONG:

Okay? This is –

PAYNE:

I’m not saying that’s how it should be. I’m saying that’s just how –

LONG:

It will fail. I guarantee you if you put Acquisition people, this will fail.

PAYNE:

But I want to put in context the way it is right now.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

VELOSA:

But to me, this and this are the same people.

LONG:

No, absolutely not.

VELOSA:

They have to be 'cause they drive the -- They don’t do it at –

LONG:

No, absolutely not. The people that understand the, that have the scientific knowledge, the other knowledge here, are not the people generating the hypothesis.

VELOSA:

They have to have the same scientific knowledge.

GRAY:

Six, I would say six data analysts there. That’s – I’m going to put a number. Six analysts.

GOLDHAMMER:

Six analysts doing passive ID?

GRAY:

Doing. Doing passive analysis and this is going into kind of what the other table had with --

LONG:

[..?..] cell biology [..?..] --

CULPEPPER:

Nah. I think you could do that, I think a team of three development oriented, totally development oriented.

[Simultaneous comments]

GOLDHAMMER:

It’s a small number of people but are there partnerships there as well? You’ve got to be doing all kinds of – or is this all coming from inside the government?

GRAY:

No. I think there’s partnerships. I think there’s --

LONG:

See this arrow here? That’s the [..?..] and that’s the CIA and that’s the DIA.

GRAY:

Right. I think you’re getting input from analysts that are in other communities. I think you have another, you know, another two people that are doing the data mining of different online systems, you know.

VELOSA:

Right. But you then have to have the hypothesis generation and you have to have folks with an ability to understand the technical issues as well as the objectives from the questions. And to me, I’m not sure I understand your logic about why you wouldn't want them to then be involved later.

LONG:

I don’t want them evaluating their own hypotheses. These guys, their job is to think up scenarios which may be a – you know, in IBM we call them “wild ducks”, okay? These guys think up things and we have this whole set of here, you know, all of these guys here that take these

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

 

hypotheses and say do these make any sense. You know, your job is to be a visionary and think of things that are possible.

GOLDHAMMER:

We’re not going to resolve this now so Paul, I think Paul has a comment.

SAFFO:

I wish Harry were here 'cause he would support me on this. The secret sauce here in the Valley for identifying technology disruptors is really simple and it’s called lunch. It is, “Hey, Darrell, let’s do lunch” and we talk. It’s lunch at the sun deck, it’s social events and the biggest problem I see that the government has is everything’s a goddamned pain in the ass to do in terms of, you know, forums to buy someone a beer. [General laughter] You know, it’s really good form. And so I just, I throw this out because what worries me as we start getting down to this, it starts getting formal. And the top-level thing I would say is this is a small team of people total. I once had a secretary who was a born-again Christian and I was complaining about the politics in my firm and she said, “Hey, look at Christ. He had 13 partners and that was one too many.”

[chuckles]

 

LONG:

Ouch.

SAFFO:

And so I would say, I would go the hypothesis you could do this entire thing with 12 people.

GOLDHAMMER:

So I hear a general consensus that this is not a lot of people.

GRAY:

No, it’s not a lot of people.

GOLDHAMMER:

We can decide whether it’s the same people or not, we can solve that, but it’s not a lot.

VELOSA:

Okay, so we have a number here of 12 total for everything.

GRAY:

I could go with 12, I could go with 12.

LONG:

I could go with 12 too.

CULPEPPER:

I could go with 12. And so the question is, you know – and I think you --

SAFFO:

Or you can put 12 or less, 12 or less.

GRAY:

And I think you divvy it up, divvy the 12 people up, based on what you’ve got going on at the time, what’s coming in that you need to have done.

SAFFO:

It’s a small team of people and it’s in the NAS model which, you know, let’s face it. The National Academies, the way it does its work is the same way that, you know, Blanche Dubois in Streetcar Named Desire. “I have always relied upon the kindness of strangers.”

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

GOLDHAMMER:

Are there any specific partnerships that you see as being critical to doing this with specific organizations?

CULPEPPER:

Search engine. I would go search engine big time.

CULPEPPER:

Search engine on this side. I think this is --

VELOSA:

But to Darrell’s point, the CIA -- I mean, it already exists.

GRAY:

So search engines on that side. On this side I think you’ve got social networks, on this side I think you have social networks, I think you have special interest blogs, you know, so --

CULPEPPER:

This is like cloud shields and like –

[simultaneous comments]

LONG:

Yeah.

GOLDHAMMER:

And then what about up here? Is there anyone, is there – you know, Stewart had the like, you know --Is there anyone, is there – you know, Stewart had the like, you know –

LONG:

This is the big boss.

GOLDHAMMER:

This is the big boss.

LONG:

Yeah.

GRAY:

This is, as Ken was saying, this is, you know, this is –

LONG:

You know, the Customs Service, right? It’s interested in --

PAYNE:

I’m staying out 'cause I got too involved earlier.

GRAY:

Okay, yeah, you got too involved earlier. No problem, no problem.

CULPEPPER:

I think there is something else over here though, potentially. I call them gravity wells. They’re big, they’re portals that people are drawn to because they’re connected to other people, right? And, but I think the long bet or the Long Now Foundation is actually, I’m not going to say it’s the right one but it has many of those characteristics.

GOLDHAMMER:

For up here?

CULPEPPER:

Yeah, for up here. It bears that fabric, right?

BRAND:

At Long Now we continue to be astonished by the impact.

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

CULPEPPER:

Yeah, yeah.

VELOSA:

And I would also argue that even the portals that connect them with the feds, the State Department and all that there, that would also have here --

GOLDHAMMER:

This is an interagency, interagency?

VELOSA:

Yeah, interagency and sort of --

LONG:

Well certainly there’s, you know, there’s communication, right?

GRAY:

Scientists don’t do reality.

LONG:

You don’t want everybody doing the same thing and coming up with different answers and not telling anybody or you don’t want to build another set of stovepipes.

GRAY:

How many perfectly good hypotheses were killed by a collection of data?

GOLDHAMMER:

Any other – just look at the table quickly. Any other resource requirements that you think are important before we wrap this up? We have a small number of people, we’ve got –

LONG:

The hard thing that you’re going to have dealing with this is inter-agency cooperation, okay? This one, it’s a nice green arrow but this is hard, right? You need to convince NSA, you need DIA, CIA, to give me the information that I can look at to try to find the signal, right?

GOLDHAMMER:

How many people are cleared in this system? You play in that world a little bit.

LONG:

It depends on who this is, right? If this is the Customs Service it’s different than if this is DIA.

GOLDHAMMER:

But no, but since -- I mean, hypothesis generation was your idea. One question I have for you is like how many of these hypotheses are classified and get stored in a vault versus –

LONG:

It depends on where this information comes from. If this is all open source stuff, then these guys are all unclassified. If this came from HUMINT then --

GOLDHAMMER:

Then it’s highly classified.

LONG:

-- it’s highly classified. So it really depends on the inputs here.

GRAY:

I mean, any, yeah, 'cause any public data mining and stuff like that is [..?..].

[Simultaneous comments]

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×

LONG:

If this is just going out to the Internet and reading blogs and, you know, whatever, then this is completely unclass. If this came through, you know, interesting ways then –

GOLDHAMMER:

Any other final comments and we can otherwise head back to the table? All right. Thank you, guys.

Feedback to workshop participants in Appendix D.

Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 157
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 158
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 159
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 160
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 161
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 162
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 163
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 164
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 165
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 166
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 167
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 168
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 169
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 170
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 171
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 172
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 173
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 174
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 175
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 176
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 177
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 178
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 179
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 180
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 181
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 182
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 183
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 184
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 185
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 186
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 187
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 188
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 189
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 190
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 191
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 192
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 193
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 194
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 195
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 196
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 197
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 198
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 199
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 200
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 201
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 202
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 203
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 204
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 205
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 206
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 207
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 208
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 209
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 210
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 211
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 212
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 213
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 214
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 215
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 216
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 217
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 218
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 219
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 220
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 221
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 222
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 223
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 224
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 225
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 226
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 227
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 228
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 229
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 230
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 231
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 232
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 233
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 234
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 235
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 236
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 237
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 238
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 239
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 240
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 241
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 242
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 243
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 244
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 245
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 246
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 247
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 248
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 249
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 250
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 251
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 252
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 253
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 254
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 255
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 256
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 257
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 258
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 259
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 260
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 261
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 262
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 263
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 264
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 265
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 266
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 267
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 268
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 269
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 270
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 271
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 272
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 273
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 274
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 275
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 276
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 277
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 278
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 279
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 280
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 281
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 282
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 283
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 284
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 285
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 286
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 287
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 288
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 289
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 290
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 291
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 292
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 293
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 294
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 295
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 296
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 297
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 298
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 299
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 300
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 301
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 302
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 303
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 304
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 305
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 306
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 307
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 308
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 309
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 310
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 311
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 312
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 313
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 314
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 315
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 316
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 317
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 318
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 319
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 320
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 321
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 322
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 323
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 324
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 325
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 326
Suggested Citation:"Transcript of Breakout Sessions for Appendix E." National Research Council. 2010. Persistent Forecasting of Disruptive Technologies—Report 2. Washington, DC: The National Academies Press. doi: 10.17226/12834.
×
Page 327
Persistent Forecasting of Disruptive Technologies—Report 2 Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The term "disruptive technology" describes a technology that results in a sudden change affecting already established technologies or markets. Disruptive technologies cause one or more discontinuities in the normal evolutionary life cycle of technology. This may lead to an unexpected destabilization of an older technology order and an opportunity for new competitors to displace incumbents. Frequently cited examples include digital photography and desktop publishing.

The first report of the series, Persistent Forecasting of Disruptive Technologies, discussed how technology forecasts were historically made, assessed various existing forecasting systems, and identified desirable attributes of a next-generation persistent long-term forecasting system for disruptive technologies. This second book attempts to sketch out high-level forecasting system designs. In addition, the book provides further evaluation of the system attributes defined in the first report, and evidence of the feasibility of creating a system with those attributes. Together, the reports are intended to help the Department of Defense and the intelligence community identify and develop a forecasting system that will assist in detecting and tracking global technology trends, producing persistent long-term forecasts of disruptive technologies, and characterizing their potential impact on future U.S. warfighting and homeland defense capabilities.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!