Archives

IPv6 Working Group session
6 November 2014
9 a.m.

MARCO HOGEWONING: Good morning. We'll give you a minute or two to come in and come to their senses.

I don't think I have ever seen an IPv6 Working Group that empty. Is this the timing or is this just because you all think we're done?

Good morning, my name is Marco, I am one of the IPv6 Working Group Chairs. There is a lot of them today. The other one currently is Shane, we'll introduce the rest of the team later on during this morning.

Like I said it's a bright and early nine o'clock for your convenience, I use the dark slide back just not to force too much bright light on you.

What are we going to do? Well it's the IPv6 Working Group. The administration bit. This session is live broadcasts, archived so if you have a question or a comment, please use one of the microphones and please say who you are.

We have also hopefully got somebody doing the remote participation, somebody is monitoring, and Mirjam, I believe, is making the minutes so we have got a record of what's being said.

Other administrative bits. Well, the meeting minutes have been cycled on the mailing list. I haven't received any comments yet. Now is your last chance to speak up. Was there anything missing or wrong in the minutes, then please raise your voice right now, otherwise ill declare them final and we'll have them published. That's done. The agenda. That's on the website. Thanks for all your contributions.

It's like I said it's still early.

What are we going to do today? We split it in three parts. We are going to kick off with some presentations. Jen is going to introduce well some of the work I have seen before, you it's sort of of related what she presented earlier on, the research on extension addresses, looking at IPv6 fragments. There is a presentation by ‑‑ not Natalie as was mentioned in the agenda, but by Colin, one of my colleagues about some stuff they see with the RIPE Atlas anchors and there is a short update by Tahar Schaa, who works for the German Government, she is going to give us an overview of how they are doing the IPv6 deployment and her national IPv6 number plan.

There's another little document Working Group input and output I have called it. You have seen it hopefully on the mailing list. Jan Zorz is going to present it but it's coming from the BCOP Task Force and it's a document about help desks. Finally we have got a whole lot of administrative bit and probably the most important part is that hopefully by the time this session ends, you won't be listening to me any more because I'm no longer a Working Group Chair and Shane. We are going to replace ourselves with a new team. After that there is going to be a bit of of a discussion on how to replace Working Group Chairs and yes, we are going to do that in that order. So we are going to first swap the chairs and discuss what's up next.

That's it from me right now. I think it says it there.

JEN LINKOVA: Good morning. I I don't feel excited why we have so many people. I don't know why we have not yet done with IPv6.

Anyway, I apologise I'm going to talk about very, very, very boring stuff. I'm going to show you some numbers, even some graphs, fortunately no formal ones at all.

What's the problem? The problem is there have been some discussions about IPv6 protocol in general and extension headings in particular. I personally found them quite cute, nice idea. However, reality sometimes introduce some challenges. For example, we suspect that currently you cannot use extension headers even if you really want to. In most cases, current use case for extension heading it's fragment header, but what if I have some interest in how to solve some of our challenges and now about using some of extension headers, can I actually do this, or probably my packet with extension headers never reached the destination, also some might be dropped. Because I do know that some networks do filter extension headers on their edge. So the question was, how big, actually, is the problem and if operators or network administrators do filter extension headers, what are actually this filtering happening? Is it happening on the source or destination network which means if client or server owners might want to use extension headers, they could do something about it like change the security policy, or is it happening somewhere in transit and there is no way we can fix the whole Internet?

So, I'm not the first person doing this. You might find some other work on this area, mostly for fragment header, so, I was curious what's happening with extension headers which are currently mostly unused. Why, by the way, people might do this filtering? It might be either because some hardware does not allow you to look after the first header after basic IPv6 one, so, for example, if you receive normal IPv6 packet with just basic header and then a protocol you can say in your firewall filter I want to permit TCP. In some hardware, fortunately, if there is extension header present your fire code couldn't recognise if it's TCP or not. All it could do is find out, yes, there is some protocol after basic header and I have no idea what is inside. So, it means those packets might be intentionally or unintentionally dropped. Or it might be some bugs in the software and hardware. So, indeed, I wouldn't be surprised to see some drops. I was just curious to see numbers. So some numbers.

First, indeed there is one large scale measurement platform we all know, it's called RIPE Atlas. So, I decided to use it because I'm crazy, that's quite simple. Use the RIPE Atlas. So I looked in the Alexa 1 million website list. I resolved those few names which you can find there in AAAA, and run a ping test over v6 so make sure those actually arrive and then I excluded addresses which belonged to my employer, because I knew the result. I actually used them for control tests, to make sure all my scripts I work on as expected. But for a real test, I excluded all Google address space and then because 23,000 addresses would require may would many RIPE Atlas credits, I choose one IP per /32 to ensure all my destinations are more or less evenly distributed across IPv6 Internet. I also knew I had ‑‑ I found some really funny addresses people sometimes put into DNS for some reason. Like, why would you put NAT 624 in your public DNS, why would you put link local in your public DNS? I don't know. Indeed you might just because you can.

So, it was destinations for my test. Then I chose RIPE Atlas probe using something similar, so, I tried to get evenly distributed cross all regions probes, picked up probes which could actually ping Google over v6 and tab with 500 probes. Does that mean I run so many tests because indeed, if you try to create a measurement with RIPE Atlas and you request 500 probes, you might not get all probes you asked for. The actual number of probes allocated for your measurements might be lower.

So, I did it initially for ICMP and we presented results if and recently I decided to see if there is any difference between ICMP and UDP in this case. So for both protocols I ran one control measurement, just a normal trace route, and then I ran 9 actual tests using hop by hop destinations option and combination of two options ‑‑ two extension headers of various sites.

So, then indeed I discard all measurements which failed normal trace route and I discarded all measurements when particular probe did not send all 10 sub tests to destination because I really wanted to see from this particular, for this particular source destination pair, what the trend is.

So, surprisingly, I can see some similar origins result despite the fact between UDP and ICMP tests we have like four months difference. What we clearly can do here, I don't know if it's clear enough on this slide, that for first of all, for ICMP and UDP, for small 8 byte extension the drop rate is much lower than I expected. It's 29 for ICMP and 20% for UDP. So I might have some hope here, you know, it might one day. Indeed if you introduce hop by hop, the drop rate increase and if you add more than the one extension header, most of the packets are dropped, which is actually not surprising because most likely most hardware, most routers couldn't recognise what's inside.

And again, as soon as you increase it the length of extension header, the rate drop increased dramatically. I suspect it's related to the fact that there is a lot of hardware and routing platforms which could not look after those 64128 bytes on the packet. So basically for such hardware all those packets are some packets are of some strange unknown protocol.

Then the main question: Where exactly packets are dropped. It's quite hard to find out because, even if you look in the IP addresses, it does not tell you much because it might be Internet Exchange IP addresses or in the period cage, you actually don't know if router with interface of this particular IS number address space belongs to ‑‑ which of the two ISPs are peering to each other this particular router belongs to. But at least I tried to get some numbers. So what I did, I basically converted the trace route hops into kind of AS path for a packet and then IP looked if I see destination AS on the path which means that packet was dropped somewhere in the destination network. Or if I could see all ASs except the last one which might means either destination AS dropped it or it was dropped somewhere close to the destination, folks on the peering cage. Basically the assumption was I think most likely people do filtering on ingress not on egress, again it's just an assumption. And then I basically compared as paths for control successful measurements and for measurements where a packet were dropped.

So, the first thing I found is RIPE Atlas, at least at the time did not work exactly as expected, there were two issues if you see two ‑‑ if you have two headers in the packet, you might not see actual inter meet hops in the trace route and secondly for ICMP RIPE Atlas version 3 calculates incorrect packet check, so packets might be dropped by destination as well.

So, I excluded ICMP tests with two headers in the RIPE Atlas, at least currently. I'm going to redo some measurements and see how it correlates with the result.


Nice thing for UDP packets I see that most packets have been dropped at the destination edge. So it's either the destination itself, the destination node itself or it's somewhere pretty close to the destination in terms of as paths indeed as path wouldn't tell be administrative boundaries actually, because it might be that even more than one AS belongs to the same administration domain and fire wall policy might be controlled by one group. Or, quite the opposite, even servers located in the same as might have completely different teams of the firewall.

For everything else, at least half of packets I actually dropped close to the destination, which again good thing just imagine for a second, that we might want to use this header in future, it means that people who control it more or less the servers might peer that kind of packets into their networks. As expected it's longest youen cries the header lengths the more packets getting dropped somewhere on the path. About half of them, half of packets with long extension header lengths actually dropped somewhere in transit. And again, some of those packets might be actually dropped at the destination. I just attributed the particular dropping node to the destination as.

So, in the origin. Not so many. And actually again, in some cases, intermediate node just doesn't respond to the trace route so I don't know actually which network the dropping node belongs to. So I suspect that the actual situation is better than I can see on my slides and probably when more packets actually dropped somewhere close to the destination.

So, I'm really trying to be quick so we can finish early and get our coffee before there is a line there.

The bad news is, it's not news actually, that packets with extension headers are dropped either because people don't care and don't put proper fire walling policy or because they drop them intentionally or they couldn't do anything else but drop them because the firewall couldn't process them properly. The good thing is for short extension headers, we might get this packet through to the final destination. And I also hope that the situation might improve over time because for some examples, old hardware could not look deeper than 64 bytes. New hardware could look like up to 256, and so on. And I know that somewhere in recently have implemented ability to look into extension headers chain in the packet. So, we might get there eventually as it normally happens with IPv6. And, again, UDP packets with short extension headers actually have pretty good chance to get to the destination. What I'm going to do next in long term.

I actually quite curious what the situation with fragment header because most measurements I have seen before were done for sending fragmented packets from clients to some central measurement node which does not actually reflect the reality. In most cases you might see fragmented packets coming from Internet, from some web server to your client. So I already have some machinery sent to some people in the room, and I'm going to see if I could measure how many packets are fragment headers dropped on the way to the clients.

And I need to test TCP. I'm actually curious if there is any difference between K PC. I hope there is not actually but at least it could be a kind of an additional verification from my results. In the end I need to run my test for ICMP with two headers because current results I have kind of unreliable.

And the most interesting thing for me is to see what the trend is. Is the situation going to change over time. Is it getting worse or better? So probably, I might ‑‑ I'm definitely going to do that, I'm not sure if there is any interest in the audience, but I'll do them, repeat the measurement in one year to see what the actual trend is and if you have any other ideas, I'd love to hear them right now or in the break or just drop me an e‑mail.

That's all I have.

SHANE KERR: Thank you.

(Applause)

SHANE KERR: So, I guess one question is you did this using the Atlas network. Did you make your testing something that other people can use?

JEN LINKOVA: I mean publish can ‑‑ like measurement results or the script.

SHANE KERR: I meant the scripts actually.

JEN LINKOVA: I'm a horrible programmer. I don't want to show that to anyone.

SHANE KERR: You don't understand the way Open Source work. We all write crappy code.

JEN LINKOVA: I'm going to publish it all right. There is no test.

SHANE KERR: I guess the reason I ask is because I was wondering if I, as an operator, because it might be interesting for me to see my results that came out of this, without having to run these tests.

JEN LINKOVA: I actually hope that operators know if the filter extension headers or not. But, yeah, okay ‑‑ as I'm going to repeat the measurements, I might get those script to the stage where I can show the source code to other people.

SHANE KERR: That would be cool. And also because half of filtering doesn't occur at the edge, you may not actually not know if you dropped these packets. Anyway ‑‑ I don't know who was first.

AUDIENCE SPEAKER: Tore Anderson. Very interesting, thanks. I have a suggestion for the any other ideas thing. I would be very interested in seeing how the ISP extension, how they'd be Teredo in the Internet. Because that has a very, very concrete and important use case, and it shouldn't be materially different from how it is in IPv4 because routers can see inside it in IPv4 or IPv6. So it would be interesting to see if the ESP extension is reliable.

JEN LINKOVA: I think I forgot a very important slide in my talk. I forgot to thank RIPE Atlas team for helping us with those measurements because they kindly agreed to implement extension header support in RIPE and they answered all my stupid questions and given me credit so I think now it's more a feature request to them to implement the ESP support, at least if we want to do it on RIPE Atlas. And I see Philip at the microphone. He might comment later.

AUDIENCE SPEAKER: Randy Bush, IIJ. Two things. I have always wondered if what got us into trouble in the first place was over reaction to HDR0. And people just saying... but just thing just occurs to me. So I have less drop close to the source, more drop as I go through the net, highest drop at the destination. So, this could be explained just by a general normal distribution of droppage. The further you go the more likely you are to get dropped.

JEN LINKOVA: Maybe. But ‑‑ yeah, maybe, I was surprised how many ISPs filtering a lot of transit traffic. But it might be one of the explanations yeah, you are right.

SHANE KERR: Did you ‑‑ about that, did you ‑‑ I can't remember, did you save the full hop count for these so you could do that kind of analysis?

JEN LINKOVA: Part of the problem is in some cases I see a response from the final hop, but I don't see any trace route response from the intermediate hops. All those hops like, I see lead local reply in my trace route. Link local source, all these stupid kind of source which I cannot use, in some case you see slightly difference in the paths between control measurement and test itself and it might be attributed to the fact that extension headers are treated differently or it's just a random load balance. Because there is some delay between you around the control measurement and your test itself. So, that's why I tried to ignore particular IPs, I tried to look in the as path to see at least, because within your network your packet might take different paths, different LSPs, whatever, you don't know. It's really hard to say.

AUDIENCE SPEAKER: Philip, RIPE NCC. Yes, if people want to test security stuff, then we can also try to add those headers. I wanted to comment on Shane's question. As far as I remember did something special for LIRs that we could test their IPv6 connectivity. So in theory, if people want it we could try to add testing extension headers to that as well. I don't know how much work it would be. I mean, when it comes to using interfaces it's sometimes a lot of work. But it's something we could do and otherwise it's up so Jen to polish up her scripts.

SHANE KERR: Well that's an interesting kind of offer.

AUDIENCE SPEAKER: Hi, Sandy Davidson from Allegro. I wonder if ‑‑ and this is quite a qualitative question rather than a quantitative and scientific question. But I wonder to what extent filtering is the use of or effects of normal out‑of‑the‑box config and how much filtering is a result of lines of actual config that's been put on the transit networks and on the destination networks because my guess is that there will be a lot of people who don't know the difference in what you may be saying there in the presentation. I am getting some networks out there have got out of the box config and that could be having the effect.

JEN LINKOVA: Yes, okay. I am pretty sure is it part of drop to destination might be because destination node just ignored that packet. It might be like host behaviour. Second part with for example a fire wall on the router, so, firewall devices, you say, I want to peer trace route packets with high port, or I permit ICMP. However if your hardware doesn't look deep enough it doesn't manage the firewall, right, because some have just implemented the ability to specify I want to watch what upper layer product and what extension header, at least the number protocol numbers because extension headers are and that parallel protocol shares the same protocol address space. So, it might be just part of your hardware ability or it might be explicit filtering, I don't want any extension headers in my network because I don't know what's inside.

AUDIENCE SPEAKER: Two things. First, I'd like to add a comment to Sandy's question. Actually, the default behaviour of most enterprise firewalls now is to drop all extension headers, pretty much like firewall 1, for example, they drop everything except for ESP and AH, so we did some testing and that is the correct observation that default behaviour is to drop most of them.

Second, looking at your conclusion slide, it occurs to me that there is not much production use of extension headers in the Internet right now. This would mean that, say, extension of protocol development involving extension headers will not happen. That is why I strongly suggest to deprecate extension headers at all.

JEN LINKOVA: We just even in a killer app.

AUDIENCE SPEAKER: Which is not going to be developed. Why would you do that if you have to expect that 50 to 80% not Internet drop it anyway.

JEN LINKOVA: If we do filter it in the closer destination and I am on ISP and drop this stuff and suddenly I have a killer app and I might use, it might stop using them or I might come to my surrender and ask them I want to be able pyramid particular pack package with particular extension headers, please let me do this pause I need it.

AUDIENCE SPEAKER: To best of my knowledge drop at their boundaries, most of the extension headers anyway, so why would ever anybody develop a killer app ‑‑

JEN LINKOVA: I really want to change it at least for my employer. I'm just waiting for fixing some limitation which are outside of my control. But I actually do need to pyramid some extension headers.

AUDIENCE SPEAKER: Randy Bush. As one of the early members of the IPv6 haters club, but do realise my company was the first one to commercially deploy IPv6 in '97, that, this is another one of those wonderful features of IPv6 and second system syndrome that's not working, and yes, there are things I would like out of it, but it got so much garbage added to it that people, you know ‑‑ and it's just not working. It's just like PMTU, we really need it ‑‑ I'm not ‑‑ v4 PMTU, we really need, it doesn't work. EH is not flying and I just ‑‑ do you really have hope that you're going to get it?

JEN LINKOVA: I need more numbers, that's why I'm doing this. I'm just curious. And I hope you don't ‑‑ you are not suggesting to deprecate to ‑‑ aren't you because ‑‑

RANDY BUSH: I don't need to deprecate it. The world depricated it.

JEN LINKOVA: It's not working as well as I'd like it to work. But I think to some degree ‑‑ to some extent it does work. I know you can't rely on it completely that's why the protocols must be a little more robust, yeah. I see your point yeah, but I was just curious, okay. I was curious how bad it is because it's actually interesting, you might actually use that stuff within the network, within the administrative boundary when you want to control everything right. But the yes is it hardware which does not support it so even if you want to do it within your network you probably could not. Or...

RANDY BUSH: Yeah, yeah, I would just remind you of I think I saw Garab here this week, 96 bits, no magic.

SHANE KERR: On that interesting disagreement of opinion, I think we have to end it there the thank you again.

(Applause)

So Natalie got a haircut and...

What's actually good because the previous talk was based on results of RIPE Atlas so now we're going to get into the details there.

COLIN PETRIE: Good morning everyone, thanks for coming along. My name is Colin Petrie, I am a systems engineer at the RIPE NCC. And I'm here to talk on experiences that we had in the RIPE Atlas anchor rollout project related to IPv6 and a variety of issues that we had along the way.

Now, I'd like to also say thank you to my colleague Natalie who helped me produce these slides and a lot of moral support for this. So, the hope is that this will sort of highlight some issues that people should be aware of and sort of common mistakes that are being made in IPv6 networks.

To first of all the RIPE Atlas anchors themselves, they are larger versions of the Atlas probes, they are meant to be installed in core networks and network operators in order to act as a larger measurement target for the probes themselves and to measure each other.

The requirements for the anchors, we basically said you know, the ethernet port, and they should have public IPv4 address, public IPv6 address, they should be static and the IPv6 address should be native. When they were developing this, one of the first problems that we had was the software instillation system for the anchors themselves what we did with them was we had a USB stick instillation image that people could use to deploy the systems. These were based on cent OS 6, and this had a fun little big in the installation system where you could specify an IP address, an IPv6 address and there was a single field for the gateway which helpfully in a documentation says the gateway can be an IPv4 or an IPv6 address. Which means that if you try and use both, it actually tries to use the same gateway value in both protocol families. So, depending on which one ‑‑ which gateway you actually fill in there it will explode one or the other protocol families during the installation. Which is a bit of a silly bug. It's actually fixed in the next release of CentOS that came out this year.

However, the in the CentOS 6 release is still supported until the year 2020 so that's still going to be around for quite sometime.

This was actually why with the installation system we ended up having to resort to deploying over IPv4 and then remotely logging into the system and configuring the IPv6 later.

The boxes: The way head hat sets it up the boxes come with IPv6 enabled for SLAAC originally, so we basically log in and disable that and push the manual config to it.

So, once we then pushed the config to it ‑‑ or once we had connected over IPv4 to get into the box, we then started to come across a variety of different issues that we found with the host networks that were there. There was a bunch of different classes of problems. Some of which came up multiple times. So these are sort of common mistakes that people might make and common issues that people should watch out for.

The first one that we had, when we logged into the box, it had come up using SLAAC, it had IPv6 connectivity, and it was getting a router advertisement to it and the gateway address was link local. That worked fine. We pushed the static config to the box and it stopped working. What was then happening here, it turned out, after we looked into, it was that we weren't getting any neighbour discovery basically from the global address that we had been given as the gateway. The link local address that we were getting on the router advertisement worked okay but the global one didn't. If you are using Linux, the ndisc and rdisc, these are useful for testing neighbour and router discovery for debugging. As you can see we are not getting a response from the global address. However, from looking at the traffic on the anchor, we were still getting the packets. They were arriving at the anchor, but the anchor couldn't send any because it didn't have a neighbour discovery entry for its gateway.

Now, it turned out that what the issue here was, was you have in the gateway address that we have been given here, you have got a v6 prefix and then they have got an addressing scheme where the v4 address is embedded in the end of the v6 address. Which is something we have seen as the hosts use sometimes, and in this instance that was the gateway address, but what happened is this this host was using a couple of different naming schemes, or sorry, numbering schemes and, sometimes they used the ::1 address as the gateway. In this case they told us one and then configured the other. So they had two different numbering schemes and basically they were getting confused between them.

So this one got fixed by them just adding the address that they told us as the gateway. And then it was all working.

The next example was we logged into the box, it got working IPv6 after we pushed the config to it. We could ping and trace route over ICMP. It turned out we couldn't get any TCP or UDP packets to and from the box. What had happened here after we spoke to the host was that with ‑‑ the requirement that we had of having unfiltered IPv4 and IPv6 addresses, they had unfiltered the IPv4 ones and forgotten to do the IPv6 ones. So, they got back to us and said yeah, yeah we have fixed the filters and then everything worked.

Another example that we had here was one that actually confused us a bit as well as the host. The host gave us an address for the anchor which was the all zeros address on the sub‑net. Now, we configured that on the router ‑‑ sorry on the anchor and found that the gateway address, which was the ::1 address on that sub‑net, didn't respond to any packets that we sent that as a source address. The reason for this is that all zero source address on a /64 is the sub‑net router Anycast address, and it turned out that their router was basically dropping that as an invalid source address on the network. After we spoke to them we basically got a move to the :: 2 address and everything worked fine.

However, on neither us nor the host noticed that the ::0 address wouldn't work until we tried it and then we found it didn't work at all. It turned out if you try and actually configure that up on a for example, on a Juniper router, it tells you networks you can't do that, that's silly. But, the Linux boxes don't stop you from doing it, so you can configure up an address that just won't work.

This one was the most common class of problem that we came across. Which was basically when we'd log into the machines, because they were originally set up to use SLAAC, they were receiving router advertisements. And we'd log into the system and find there was a whole pile of errors in the Linux kernel logs complaining about invalid router advertisements on link. You see the resulting prefix that we get back it a /56. Basically that doesn't work because on an ethernet interface you must use an ethernet 64 bit prefix length length and we saw this for a variety of different networks and a variety of different sizes. In these cases, no almost all of these cases that was the size of the sub‑net that the host had actually configured on the link, which works fine if you configure it as a strategic address but they were also sending router advertisements for the same sub‑net size, which is inslid if it's not a /64. In one particular instance they were advertising a /128 on the link which is pretty useless and actually the /128 prefix was actually the gateway address itself, so in this case they had configured the gateway's IP also in wherever it is in the router configure that advertised a router advertisement or something like that. These were not actually problems that stopped us from doing anything, but because we push a manual config to the box, however it was a very, very common one, and one of the ones that you know, it sort of indicates that that's probably a sort of maybe some routers from a default setting that sends router advertisements for the link even if it's not a valid one or something like that. But this was a very, very common problem that we saw.

This one here was, we logged into the system and we found that we could ‑‑ when we tried to ping our trace route out from the box the first hop on the trace route gave us exclamation mark N as a response in the trace route which is the ICMP destination unreachable code. Nice of them to actually send it. But what it turned out was after we contacted the host, they had forgotten to add a default route on their router. Again, IPv4 worked fine, but it was a major thing to miss out on an IPv6 config.

Then this example, took us a little while to sort out. The request that we had for must have native v6, it showed up on this one that they were using some form of tunnel somewhere several hops up on their network at the anchor itself, it was just ethernet, MTU 1,500 quite happen low, and but what was happening was there was a path limit along the way of 1480, which is usually a sign of a tunnelled network somewhere along the way. But they were blocking ICM v6 packet too big, so the MTU past discovery stuff fails completely and the anchor keeps on trying to send 1,500 byte packets and they just get dropped on the floor.

We contacted the host about this one, they ended up actually renumbering us on to a completely different network, because they weren't able to sort out whatever it was that they were doing on the network, they couldn't unstop it from blocking these packet too big messages. So they decided here is another network where it works and moved us on to that instead.

So, what we're sort of looking at here is still still a lot of issues, both in current main stream software releases that are still going to be with us for quite sometime. They are still supported OSs, and there is quite a few commonalities in the mistakes we see, sort of certain patterns that repeat on several occasions which are indicative of various sort of people not quite fully having their provisioning systems worked out, not quite understanding it, you know, for getting to do v6 as well as v4, all that kind of thing forgetting) we can learn from this, there are things you need to keep an eye out for, or you can put into your training, or things like that. And we thought we'd share this and hopefully it's useful to everybody.

And that's the end of the talk.

(Applause)

AUDIENCE SPEAKER: I have a comment, not a question, because I have been ‑‑ when you say IPv6 is not as understood as before, I actually suspect v4 is not understood well enough, we just like it to have default configuration races work and have all this in place, you can copy and paste.

COLIN PETRIE: I very much agree.

SHANE KERR: You are saying v4 is not as well understood as v4. Okay. Thank you. Interesting.

Okay. Our next presentation is about an update about v6 rollout in the German Government.

TAHAR SCHAA: Hello. I'm speaking here today on behalf of the German government. I'm a consultant for German Government and Swiss Government, now I have the German hat on my head. And I want to bump up the research development project for IPv6 in German public administration, and there are some results which you should take care of.

Why did we do this? We had some urgent support needs. We needed some real detailed specs to procure the components and network connectivity and we needed some best practice because not every authority can develop them for themselves.

And of course, we need ‑‑ there was an urgent need of training material because no one had a clue about IPv6 in the public sector in Germany in the beginning, and after we had the addresses and made some training, we had to make up our mind about how it will work. So the routing has to be planned.

An afterwards we made up our minds about the other things, IPv6 could influence and we identified that the Internet thing is a main issue with this.

So, what what are the results? We created and IPv6 profile with detailed options and RFC descriptions, what do we need and which component where IPv6 is in, and we made a huge chart in Exel and now the public administration has a guideline how to procure these IPv6 components. We describe this to make them know how to use it, and in addition, we build up a typical IP infrastructure of authority and migrated them to IPv4 /IPv6 dual stack and from that we quote down an IPv6 transition guide, and in addition we made some work up slides to make them able to make their own trainings, and some checklists that they have some guides how to ‑‑ what to consider when I transit to dual stack. Everything of this is available for free in the Internet, for you also, and it's under the creative common licence.

Why you should care about it. Two of the results are now available in English, so feel free to read it, to make comments on it and perhaps it helps you.

Where to find? This is an URL where to find it here, and there is a little point, best practice, and when you click on this, you see all the documents the Germans and the English ones.

So, what did we do also? We took care about the routing, because with IPv4, in German public infrastructures NAT was very popular, so everyone had its own address space and then NATTed into the other networks, therefore now we have to have routing what wasn't there before in real life.

We had some concepts, but we didn't know if they worked. So, we thought about it, a good idea to assimilate. So, this is our simulation environment. First we thought about how to do it. We chose GNS 3, and this is the same infrastructure we assimilate in GNS 3. These are 32 routing nodes, and we loaded into this simulation some different routing policies things, and looked up what's going on. The main issue for us in public administration is that we have mostly two connections, one internal and one over the Internet, and of course, what we want to have is that internal information and communication between some ‑‑ two authorities will not go over the Internet of course, but over the existing internal network.

So, we made some simulation and showed that it will work if we load here and here the right routing policies, and therefore we can show there are some concepts which prevent from transport the information over the Internet.

What we thought about what is the future of IPv6 and we thought the next revolution is not that normal computer you have on your desk will use IPv6, of course it will like that, but the majority of systems in the future using IPv6 from our perspective will be somehow lambs, door systems, heaters and so on, because there are so many devices existing and when they all migrate to IPv6 communication, they'll be the majority. So, what about security? What about concepts of using them?

And so, we made a reference architecture, so that we have a clue in which framework, in which environment they'll be used. This is the thing ‑‑ this is the old one where we have normally gateways for these propriety systems the and today, and in the future, we'll have a router here and every node will be accessible directly via IPv6.

But we have some different systems we have to consider. We have this really small with only a small battery, they are not able to implement a full IPv6 stack with full security and so on. Then we have some reduced systems with limited power in it. And we have these systems, it doesn't look like a real computer, but it is one.

For these systems, we made up also a profile. This is not published yet, but will be hopefully next year, so that we have a profile for these untypical IPv6 systems used in the industry and in the automation environments. And the problem is while we made a migration guide for normal office systems, we had to migrate from IPv6 ‑‑ from IPv4 stack to an IPv6‑IPv4 dual stack. This is quite easy we learned because in the automation environment, we have several stacks which are really crowded and not really standards and not really documented and it's a huge mess. So we have a much higher effort and thing to migrate, because it's ‑‑ the place we are starting from, it's much more complicated than with IPv4 only.

So, I only had a small slot, thank you to Marco. Any questions?

(Applause)

AUDIENCE SPEAKER: NRI ERW, just a question given there was that example with the small ‑‑ with a /48, did you consider the impact of LIRs or providers performing strict filtering on that prefix size from the German PA space? And what is your approach to solve this potential challenges?

TAHAR SCHAA: Yes of course, we don't plan to make the world happy with small snippets of IPv6 announced in the world. We have a concept for it but it's not published and it's not enough time to describe it now here. We can do it in the coffee break. But there is a concept to do this, but it's not published yet.

AUDIENCE SPEAKER: Tore Anderson. I have a question about the procurement document. I was wondering, when the German Government goes out and buys a service or sets up some sort of service for the public, do they have a requirement to use IPv6 or is this optional or is this not mentioned at all?

The reason why I'm asking is that in Norway we have a "should" requirement for quite sometime. And it's been universally ignored by the Government. So, we're trying to make it a "must," and I was wondering if you have some insight or experience in how to go about doing that or...

TAHAR SCHAA: We really would like to have a "must," of course, especially me, but we are living in Germany; Germany is a free Federal State so by constitution we can't force all the Federal Constitutions to use it because you can't force them for anything. So, this is a problem. But you can make an offer which makes sense and then it's really likely that they use it, and this is the case. So, I have been asked when this profile wasn't already translated into English, from town Kier, in the north of Germany, we need this urgent in English because extreme networks, we sent them this procurement chart and they got a mess because it's in German. So, they use it and that's great.

SHANE KERR: Okay. Well I say this is really great and I really appreciate you sharing it with us.

TAHAR SCHAA: Perhaps one last sentence. The title of the presentation was the ongoing development of IPv6 environment. One last word to this, there is one governmental network rolled out over the whole Germany which is now able to make full dual stack and it's used, and perhaps interesting, the one who is using it at most and really strong is city of Munich. So it is used, it is rolled and so, yeah.

SHANE KERR: Thank you. And now we're going to get an update on a document being worked on by the BCOP folks.

JAN ZORZ: Good morning, my name is Jan. I have a report here on behalf of the BCOP Task Force, I will talk about the IPv6 trouble shooting for residential ISP help desks. This is a document that should remove the next barriers of IPv6 deployment.

We presented this in this Working Group last time. Since then, there is a Version 2 I believe Working Group Chairs send this to a mailing list and I hope that you read the document. So, who in the room read the document? Quite some. Thank you.

So, just to refresh the mind. + we put together quite a good group of people. Good picks experts from Europe and from North America, because we heard a lot from the operators that the next barrier is their help desk, because they don't know nothing about IPv6 and they are afraid to deploy IPv6 to the end customer because they fear that their help desk would just burn‑down in flame if somebody calls in and says hey, I have the IPv6 problem.

So, this is a starting point that Help Desks can use, adapt. Some of the people send this document to their help desks, and experimentally run it. We received lots of comments and feedback, and we implemented these comments into the second version of the document. And also, Jen, thank you Jen, thee made a work flow that at the end describes fairly quickly how the trouble shooting procedure goes.

So, what does the document contain? It's a back browned information, then it's the IPv6 trouble shooting using the test IPv6 dotcom. Thanks to Jason if he is letter for building the special version of test IPv6 dotcom. It's quite easy to use, your help desks are sent the user to this URL address and they receive back help desk code that is described in this document.

And what do we want from you? That's a good question. So, we gather this group of people together at the BCOP effort in North America and here at RIPE. The purpose of the BCOP Task Force is to gather together the operators to start writing the documents and to verify if this is the current operational practice and to document the best current operational practices, but here in this Working Group, people like to test or to check the technical validity and that the document is is technically sound. Also ask for adoption. We received lots of good positive comments, and in the last version, there was no violent disagreement that things are not working well, so, we have a feeling that maybe it's time to stabilise it and publish it as a RIPE document.

So... are there any comments, questions, suggestions, support, disagreement?

MARCO HOGEWONING: Can I see the hands again ‑‑ how many people ‑‑

JEN LINKOVA: A comment actually, a request from the audience, I think it would be quite useful to hear if you think ‑‑ even if you think it's useless. Because probably some people might not send in feedback because they don't have positive one. I really really would like to hear if you think it doesn't make any sense at all. Because we have had some discussions about how much could we expect from your support. So, is it too basic or it's too complicated? We really would like to hear that.

MARCO HOGEWONING: I have got Randy first.

RANDY BUSH: I'm noticing the propagation of this document, I wonder if there was a problem that it has too many extension headers on it. Nobody seems to have read it.

MARCO HOGEWONING: That's indeed my point. I'm going to take these two questions... Tore?

AUDIENCE SPEAKER: My name is Ben Gordon from the company Resilience. Given the fact that the Help Desks barely now about IPv4, would this actually do anything good? I mean, they don't invest in the IPv6, so... there is no money there at the moment, so...

JAN ZORZ: Okay. What we heard from people that want to deploy IPv6 to the residential customers is that they deployed IPv6 in the core, they deployed IPv6 in the services, they deployed it everywhere but they are not rolling it out to the end customers because of that mental barrier that their Help Desk is not aware of it. So, if we can provide them the basic tool that technical people can just print it out, take it over to the help desk, make check, help desk knows about IPv6 and what we are deploying and we can continue with our deployment of IPv6. We removed the next maybe mental barrier and also provide a useful documentation if there are troubles that help desk can use to actually trouble shoot it.

AUDIENCE SPEAKER: I don't debate your good intentions. But I think that, as I said, the IPv4 knowledge forth help desk is not there. So, if they sort of help the help desk today but not the knowledge of IPv4, I mean... they are probably not going to read this document either.

MARCO HOGEWONING: I'm going to cut this debate short. And yeah, I'll take Jen's comment and the final question, because we're sort of getting where I want to go.

JEN LINKOVA: Let me quickly comment on this. If you look at this document, because we assume that help desk know basically nothing, the whole idea was to let them know where they just escalate to the IPv6 team or if it's not an IPv6 problem at all. If you look ‑‑ if at least in some situations you can easily see okay, it's v6 problem call your IPv6 expert because probably deploying something new you have separate escalation Kline. Or everything is working fine on IPv6 and it's just a DNS problem.

AUDIENCE SPEAKER: Tim. Just a few comments. I really want to support the work. It's really great work and for us, it worked a bit the other way around. We have a monitor in our lobby where the coffee machine is so when colleagues coming in and drinking coffee we have some nice graphs over there, and at one day I put the IPv6 test on there and it's really nice that you get points how good you are in IPv6 deployment and we had 9 out of 10 points. And then my colleagues who were not that into IPv6 and started to question why have you only 9 out of 10 points and then we got into a really good conversation and you also see why you don't have the 10 points and then all of my colleagues, you know, then it's a challenge, we want to have the 10 points. And so, this IPv6 test and then also the pest practice documents got the conversation within the company flowing, and at the end all of my colleagues were involved into IPv6 and so, for me, that was one of the initial things to get the work flow running, so, I think it's great work and it's a really good starting point for in the company and then maybe spreading out to the customers, but first you need to get it in your own company. And that was the initial thing for us. So many thanks for all of the people who worked with it.

MARCO HOGEWONING: Thank you. Which actually brings me to procedural, I gave you on the head up beforehand and as we saw not many people read this document so it's kind of useless to discuss the content here. I believe that last Monday on the BoF, there was another discussion on this document, and there is some content being reviewed there. So rather than who read it, who is going to read this in the next two, three days, let's say on the flight home or something? Can I see a show of hands? At least ‑‑ okay ‑‑ that's a good thing, so please also actually do that and provide Jan and his authors with feedback via the mailing list.

The second thing is and we already got one voice of support and I know we're not going to do a formal vote and I'm not good at humming, so a quick show of hands who says yes, please keep this in the Working Group and please work towards publishing this as a RIPE document on behalf of the IPv6 Working Group. Can I have a quick show of hands? I think ‑‑ is there anybody object to go that course? Cool. Then I think we can at least say we have got consensus on the road to follow.

Then what I would like to do, and then it's a bit weird because, yes, I am stepping down as Chair so I'll take directions from the new team, but I would suggest that we leave this open for another like two weeks for comments and after that, the new Working Group Chairs will have a chat with the others, Jen you can talk to yourself, and decide on whether there are further revisions necessary or whether this is deemed fit for a final call and publishing as a RIPE document. Just to clarify, this document does not contain any policy. So there is no need to invoke the PDP on it. If there is consensus within the Working Group saying, yes, this should be a RIPE document, it can be simply be formatted and published. So, if nobody objects to the content and everybody says like, yes, it's useful enough to publish, then I would suggest to just go publish, but the final decision won't be mine. It's the group's. So, thank you Jan for your work here, keep it up and also I appreciate it it from the BCLP to take it to the Working Group as requested. And good luck.

Which brings me to the next point. Thank you. As we announced last time, David stepped down and Shane and I decided it was a nice moment to also leave the scene. We are talking a bit about how to do it. Clearly the end is near, we tried to find a fat lady. Couldn't find one, so clearly there ain't no fat lady singing, it ain't over yet.

Looking back at five years doing this, we got in pretty much in between Portugal and Prague, May 2010 was our first meeting as Working Group Chair, I believe. The state of the world at that point, well, there was this shimmer of life, this tiny little spec at the horizon, and everybody was asking themselves is there light at the end of the tunnel? Maybe there is, maybe there isn't.

I tried to look up a graph, fortunately Geoff wasn't running it, we discussed it with Google. Clearly IPv6 deployment was in the four digits mark, four digits behind the comma at that point. Was there light at the end of tunnel or was it actually light of an upcoming train? Everybody was asking them that. There is some good news. If we look at the state of the world right now, yes, we do see IPv6 deployment in two digit figures. Fortunately, we also see IPv6 deployment in the two digit figures below the point. We're not done yet. And we, because of course we will remain active in this Working Group, although we won't be wearing yellow badges.

So, still some stuff to do. And, as one final comment strong to mind, yes, we have been talking about this for quite a bit now, and although we do see deployment, there is still a long way to go. So please a little more action, and if you know your music, you know it was not a fat girl, it was a fat man singing this.

Well last time we called for volunteers and we were happy to see that there were like five or six people stepping up. Deliberations behind the scene amongst candidates had led to three new volunteers to take over our job. This is the new team if I can get everybody up here. It's Jen Lynn Cova, you have seen her present. Benedikt stock brand, and Dave Wilson, all thee I think well known figures. As you saw on the mailing list, we suggested that they are going to be appointed as Working Group Chairs as volunteers to take over the reins for this group. So, that's what we're going to do right now. There was clear consensus on the mailing list. Thank you all for your support for in the procedure and also for the candidates.

I have got only three final words and that's good luck and thanks. If you need to reach the team, the mailing list is still active and they are all on it, so if you mail them it will end up with these three people. And well that's the end of my slide show I think.

So, how to do the transfer? Well, when Rob announced Petter change places we swap the Internet. We clearly see there is no Internet to swap because it's not IPv6 ready yet.

So the other thing to give you is new badges I guess. So, here is the yellow badges. I have got my white badge.

(Applause)

JEN LINKOVA: Careful, if you catch it you might the next Chair next time. You should be careful.

BENEDIKT STOCKEBRAND: Actually they have their dinner tickets in there as well. So... one more thing. Now that you're retiring to the more passive seat. Make yourself comfortable, enjoy yourselves, and well, I guess it's time to have a beer. Enjoy the show.

JEN LINKOVA: This is the way to make IPv6 tract I have to people. Yes. When you die, you get some drinks.

Next slide...

Because we have a few more slides for you.

AUDIENCE SPEAKER: You do know that these are warning labels and not things of honour...

JEN LINKOVA: We'll find out. I did upload slides. Stock stock if nothing else ‑‑

AUDIENCE SPEAKER: Before you start as Working Group Chairs, I think the whole ‑‑ the old chairs deserve a very warm applause for all the hard work they did, so guys, thank you very much.

JEN LINKOVA: So, we have had some discussions already shall we talk in this Working Group about transition technologies and we decided that we actually should not, it's too late so it's the very, very last time I promise you we are talking about transition technology here, we are going to discuss about how we go to the transition between Chairs to avoid all this confusion because I actually had two damages for the whole week and it was quite confusing.

So, we are trying to make the transition technology as easy as possible and backwards compatible and all that stuff.

Problem statement:
Apparently it looks like you don't want the same people holding the microphone all the time, we don't want it to be a life sentence, so it probably might make the Working Group Chair position a little bit more attractive because you know that you make a commitment for quite a limited period of time. And it's always nice to see new blood in the group and on top of that, it was a kind of new requirement, let's make it more or less standard. And when the last meeting we go in this situation when we need a new Chair, the group needed a new Chair, we did not have any procedure to do this. We actually were lucky enough to do it quite easy, because we have just right number of volunteers apparently. But we still need to come up with some solution. So, we are trying to make it as simple as possible, we are not going to cover all possible horrible use cases we might or might not run into. So like what are we going to do if we have 25 chairs and we couldn't fine candidates and so on. So, so far, what we have, we have a suggestion that we have up to three chairs. Three actually is a nice number because it's an odd number so if we disagree, at least we have a majority.
Rotation period:
So let's do it every year one Chair might step down, and so, I think we will have a volunteer to do it next year, but yeah, so probably we'll see new people on the stage in three year's time. The main problem is actually how to change or remove a Chair. So what we suggest is so one month approximately, or I think it's better to do it in advance, a Chair should announce the end of the term on the mailing list. I know that a lot of people do not read mailing list, but still for those who do, it might be nice to see not at the meeting but at least one month in advance, and indeed, there will be a call for volunteers, and I do hope we'll get some volunteers. If you get too many volunteers we'll probably lock them in a room and let them discuss it between themselves and those who survived, after this discussion, if it will be more than one volunteer we might need to come up with some procedure, maybe a vote, maybe not, I don't know, but we are open to hear the suggestions from the Working Group.

And I don't know if we need a procedure to get rid of the Chair. I do hope that we'll just get v6 implemented and shut down the Working Group before we might need the procedure. And we need your help with this. I mean with getting v6 implemented not with getting rid of the Chair.

So, as a result, we are not suggesting any particular steps for it. If Working Group somehow feels very unhappy about chairs, yeah, feel free to come to microphone or into mailing list or in person, complain about this and then, yeah, we might do something about it.

So, it's basically that's it. Please let us know if we miss something, we our professional is completely wrong and is unacceptable for the group. If there is no strong objections, we are going to put it like form of text and send it to the mailing list. Let's discuss it, but let's not discuss it for too long, so we definitely want to get a kind of last call before the next meeting. Have I missed something?

BENEDIKT STOCKEBRAND: I guess so. Now, we left Jen the job to make it look like we were just confronting you with some fixed decisions, or anything. No. But, seriously, what happened ‑‑ these are basically the results are, or let's call them recommendations that we came up with lunch on Monday or Tuesday, basically what we should explain even more than these ideas we have about this, is why we came up with these.

First thing is the way the RIPE community works and gets things done is generally by doing the right thing and not caring too much about policy or administrative procedure or whatever you want to call it. And we had a couple of things in mind when we came up with this, I'm not even going to call it a proposal really, which is we want to make sure that if something goes wrong, or if something needs to be done, it can be done and we won't have a policy get in the way. And I think that's probably the most important thing we should keep in mind. I mean, if there is ‑‑ for example, if a Working Group Chair decides I can't come to the RIPE meetings any longer because of a new job or whatever, and my new employer doesn't let me go or something like that, stepping down as a Working Group Chair is possible at any time because the alternative would be to have a Working Group Chair who doesn't do anything any more. So, these are sort of the ideas involved there, and we really want to keep this as simple as possible. The only thing is, and I have learned that from discussions with both Rob and I think Shane, about every once in a couple of decades you actually want to get rid of a Working Group Chair and we might actually have this sort of thing to sort out. We have been focusing on things where situations might get ugly and we want to establish a policy, and pretty much everything else should be common sense and people trying to get things done.

So, that's where all these thing come from basically.

Comments anybody at this point?

AUDIENCE SPEAKER: Mirjam from the RIPE NCC. Just to clarify a question, maybe I missed it. It is possible that the current chairs are also candidates for the next round or is that out of the picture?

BENEDIKT STOCKEBRAND: Well, yes, yes, we actually thought about putting a limit, like, sort of like six years or 25 ‑‑ Rob just left ‑‑ to this, but actually, well, there is reason to do that, which is that people who once get this sort of title, job, whatever, anything fancy, they can't let go. But well, at least my opinion on this is, with the sort of crowd here, I'm not really that worried about it. I mean, these things will eventually sort out themselves because people ‑‑ the worse thing that can happen is nobody will go to the Working Group and somebody in the next room take over and you have another Working Group group if things really get ugly. So that's not really what we have considered necessary. Simply from what we have seen so far, in the RIPE community, that people are sticking to the sort of job is apparently not that much of a problem that we have to worry about this. And anyway, normally the problem is you only get new Working Group Chairs if the old ones decide, by the way we're stepping down next time, and you decide who is going to do the job next. Otherwise, you won't get any new volunteers a lot of time anyway, so...

DAVE WILSON: Of course at the end of the day, what we think doesn't matter. We're only the Chairs. What matters here is what the Working Group wants. This is our first shot at what we think is probably reasonably sensible. But of course, what we need now is just to begin a discussion. If this looks good and you want to get rid of this discussion as quickly as possible and this will do, very good and that's useful put and if you want something changed, we have to change that of course.

BENEDIKT STOCKEBRAND: And we're not afraid to do that. If you are unhappy with anything like this or want some more backgrounds on why we suggest things ‑‑ these are just suggestions, this is not something about to be set in stone. That's what the mailing list is for. The only thing I was afraid of is when we first talked about this is we'd have a discussion here about wording and some sort of fixed document and we happily avoided that by just using bullet points which are probably not good enough for final procedure and we hope that people just want coffee. Because in our opinion, we have to sort out the details at some point, but that's best done in the mailing list anyway. And so...

AUDIENCE SPEAKER: Rudiger Volk. Yesterday in one of the sessions a similar discussion one or two people made remarks about what a waste of resources and energy and time it is to have six, seven, eight, nine, discussions of the same type all with fuzzy input. Actually thinking about it one more time, my suggestion this time would be if there are a couple of different proposals, why ‑‑ and most of them fairly fuzzy ‑‑ why don't we have a write‑down of the proposals, general discussion for all the mailing groups with the different versions and next time a Plenary discussion of what is the right thing for everybody to do, which leaves the option that the Working Group that has real special requirements does take its own option.

BENEDIKT STOCKEBRAND: Rudiger, I think you really got the point there that we tried to avoid bringing up. Basically we are all lazy bastards and we just hope that some other Working Group came up with a fully articulated proposal we could mostly copy what we like and leave out what we didn't. So ‑‑ but basically the idea is, because in the past there have been discussions in the Working Group ‑‑ or there has been a discussion on a RIPE‑wide policy which didn't work out to leave the discussion to the Working Groups and ‑‑ well, we'll copy from each other. These aren't copyright protected things or anything.

RUDIGER VOLK: Why ‑‑ well, I didn't know of that discussion. Has that happened bottom up or top down?

BENEDIKT STOCKEBRAND: Well, normally things should be bottom up in this place. But I have actively been involved in this sort of discussion before. So... Dave just pointed out we are running out of time, so I'd like to close the mike lines please.

AUDIENCE SPEAKER: Moss an Swissey, a very perrogative comment. I would invite Jan there to integrate in three years time in the pick up Working Group the best way to select and rotate chairs in Working Groups.

JEN LINKOVA: Congratulations, you have been volunteered, Jan.

RANDY BUSH: It is 29 after. I suggest any Chairs who let this go past 30, be removed.

BENEDIKT STOCKEBRAND: Thank you. Very quick one, Marco.

MARCO HOGEWONING: There have been been a lengthy discussion on the Chairs mailing list amongst the Chairs to come up with a unified proposals. One of the comments raised of the Working Group Chair list was indeed not the right type of venue because it was not open and transparent enough. So hence we take it to the mailing list.

DAVE WILSON: Thank you very much. The mics are closed. It is 10:30. I think what we'll do is this: It sounds like, although we have discussions about whether or not there should be a procedure and so on, I am not hearing a lot of opposition to these principles so I think so we should go ahead, write up based on this, maybe we can copy something based on this, we'll say, I assume that will go to the mailing list. Are we happy with that? It sounds like we're happy with that.

JEN LINKOVA: Okay. See you in Amsterdam.

DAVE WILSON: I just want to say again thank you so much to Shane and Marco for leaving the Working Group in such good continue which we find it and we look forward to taking care of it.

BENEDIKT STOCKEBRAND: We forgot somebody, we also like to thank both John and Edoardo who also stepped up as candidates and then eventually stepped down to make life a lot easier for everybody by doing so. So we didn't run out of candidates and we didn't have to fight it out.


Okay. Coffee break.

LIVE CAPTIONING BY MARY McKEON RMR, CRR, CBC

DOYLE COURT REPORTERS LTD, DUBLIN, IRELAND.

WWW.DCR.IE