Thank you very much, josh, for that very nice introduction. I am aaron khere to introduce te final panel of the day that relates to another important set of Public Health functions, that Greater Transparency about information and regulatory decisions about the fda can play. I want to thank allen and pew for hosting this meeting, for organizing the publishing of the special supplement and of course to mr. Gottlieb for his remarks at the beginning of the day. So the focus of this final panel today is the fact that in certain high profile cases manufacturers have released incorrect or incomplete information into the market that was directly counter to the information, the contemporaneous information the fda had. They noted that selective publication of Clinical Trial results has in the past created a misleading picture of the safety and efficacy of a product with negative implications for Public Health. And as a the practice now still stands, manufactures have wide latitude to publicly characterize data that were submitted to the fda without much risk that the fda will correct the record. I think were going to try to talk a little bit about in the next hour is, you know, in these cases does the fda have a Public Health or an ethical responsibility to correct the record . And if so, how. So in the blueprint we outlined a couple principles that might help organize this kind of approach. So first, should the agency adapt a basic set of standards for when to correct misinformation . Should the agency give advance notice to manufacturers regarding any concerns they have, and if so how would that happen . And should the fda disclose the scientific information thats the basis for its concern about misinformation. Hopefully well be able to get into some of those topics with the distinguished group of panelists we have today. Im going to introduce the pa panelists who are going to give their remarks and their academic points of view. Meanwhile, if you have those cards please do start writing down your questions on those cards so we can address them in an efficient way. Our panelist today first up is peter lurry. Previously he was the associate commissioner for Public Health strategy and analysis at the fda. And in our packets there are much longer bios for these very distinguished people. Second is dr. Erin turner. His research is aimed at increasing medical transparency to make the Evidence Base more complete, truthful and reliable. And then third we have dr. Michael curome. Health citizen is one of the leading Public Health advocacy groups in the country and editor of a monthly newsletter that provides reviews of the safety and prescription of over the counter drugs. Unfortunately our fourth panelists was not able to be here because of extended jury duty she had. So she is serving the public in a different duty today but is not able to be with us. I would like to start with dr. Lurry and please save all our your questions and well collect them at the end and hopefully have a good discussion at that point. Thank you very much. Okay, good morning, everybody. And thanks for the opportunity to speak. Aaron said that with this panel is about is the difference between information that the fda might hold contemporaneous with information in the public realm. My background had been in an activist group and Public Citizen, and you can imagine me really rubbing my hands together with glee when i was employed at the fda and finally got a chance to see some of this contemporaneous information. And the information that im going to present today is really about finding a way to make some of that contemporaneous information public by aggregating data in such a way that you wouldnt be able to identify a particular company or product but would nonetheless be able to convey certain information about this contemporaneous information. So thats the basic outline. As mentioned im now with the center for science in Public Interest to this research i will present was done while at the fda. Im speaking not as an fda employee since im not one, but some of my colleagues from fda who helped work with me on these papers are present today. So really the presentation consists of two parts. A presentation about two separate papers. One of them youve actually kind of heard a little bit about through the course of the day, a publication on complete response letters. Which for the benefit of those watching, complete response letters are letters that disclose the reasons why the fda might have decided not to approve a drug. The second paper relates to publication of postmarketing requirements or ill use the term pmrs, which are requirements the fda can now impose upon a manufacturer whose drug has been improved in order they conduct follow up studies. So those are the two studies, and they have quite a bit in common as it turns out, and ill turn to them in turn. So heres the first one. It relates to complete response letters or crls. And what we did in our study was to look at all the crls for new drug applications that had been put out for center for Drug Research for the fda for the period of 2008 through 2018. What we did is i assume is similar to something that dr. Gottlieb is having someone at fda do is we chopped them up into seven different domains like safety, effectiveness, pharma, pharma and so forth. And each was assigned to these seven domains. We actually get further into what we call subdomains, but ive left that out of this talk. And then we look at the percentage of crls that had a matching press release. Question one was whats in the crl because nobody really knew. And the second part was given that we at the fda could see the crl, what was it that the public could see . What could they see in a press release . What percentage of the statements appear in the press release . Or do they release a press release and if so, how much information was actually disclosed . And we actually look at the fcc filings to see if there was any supplementary information over there. Again, the first question was why does the fda turn down a drug . And that a fundamental question to when there was no public answer. And in the absence of information there were at least some people who believed the drugs were sometimes not approved for lack of a crossed t or lack of a dotted i. And our experience was that was not it case. It was usually for substantive reasons and we went about finding if that was true. This is the number of the crls that made mention of a reason for disapproval in these seven domains. And you can see, first of all, that there were a array of Different Reasons that the products were turned down. You can tell by the numbers here that many had much more than one reason, right, since there were 61 and many of them are in the 40s. And you can also see the efficacy, safety, and this is clinical manufacturing and controls were the primary reasons. But most importantly about two thirds of the crls had an efficacy reason for turning down the product. 80 had either a safety or efficacy reason included in the crl, and 48 had both. So we drew the conclusion from this that in fact the fda does not turn down products for trivial reasons, and hopefully that has helped to counter the misinformation in that point in a general way, of course, that doesnt answer the question of what might be happening for any particular crl because that has yet been released. Were looking at crls as a unit of analysis, and the question is what did companies do when they received their crl . The answer was that in about 18 of cases, 11 of the 61 crls, there is no press release whatsoever and really no way for anybody to know that the product had been rejected. In an additional 13, the company put out a press release that said that a complete response letter had been released but provided no details whatsoever what that reason was. And then the rest of this or the fraction of the statements that were matched. So we have 61 crls. Theyre maybe about ten pages long apiece. 61 crls had an average of 10 to 11 statements. The question is what fraction of the statements were actually there . And you can see that in those that had a press release that actually gave reasons, you can see that typically only one to 25 of the statements were matched and only occasionally were higher fractions of the statements actually disclosed. Okay, so now were changing the frame to instead of looking at the crls, now the unit of analysis is the statement themselves of which i mentioned there were 687. And now were looking at those same several domains i mentioned earlier. And in the red are statements that never appeared in a press release. And in the yellow are the statements that did. And so you can see what the matching rates are. The highest matching rates are for safety and effectiveness. The overall matching rate is 14 . Higher in safety and effectiveness, like i say, and much lower in these other things. So the companies are leaving out, well, 86 of the information. Theyre more likely to disclose safety and effectiveness information. But even their critical information is often not included. For example, there were seven crls that mentioned a statistically significant or at least an increase in mortality rates between treated and comparison groups. And only one of those seven trials had a matching press release that disclosed that fact. There was if there was a requirement for an additional Clinical Trial, which is very much, you know, the key sort of turning point for the company as to whether or not they will be able to rapidly get a product approved or not, even that was only disclosed about 59 of the time. Okay, finally we looked at the Securities Exchange commission filings to see if there was any Additional Information over there. And we did find a small amount of Additional Information, which brought the overall amount of Public Information just a little bit to 14 to 15 . But most of the information is not made available. Again, this is at the statement level, in either source. So the second study, which ill hurry through, is about these post marketing requirements, the pmrs. We look at all of those that were fulfilled. So this does not get at the action of whether or not the studies are actually done. This is among those that are done. And we asked whether or not by at least three years later, 2016, the pmr had been published either in a scientific literature, in clinicaltrials. Gov or in either source. The pmrs were published either in the Science Literature or on the website, and you can see there are higher rates in Science Literature than in clinicaltrials. Gov. These are the interventional Clinical Trials as opposed to other trials that were fulfilled. Theyre in the first place more interventional Clinical Trials than other kinds of trials. And you can also see the greater fraction of interventional Clinical Trials are published than is the case for the other trials. The first bullet point here saying that bullet point i just mentioned is a statistical difference. In the interest of time ill just pause over what the predictor predictors are there. Not especially strong or strike ing what happens. Just a conclusion, again two different kind of studies that turn out to have a lot in common. Both the letters were recommended in the transparency report as back as far as 010 and that data shows in both cases the reporting has been less than optimal. Broadly, they share the same potential solutions despite being very different kinds of documents. On the one hand the companies could, at least in theory publish voluntarily. But they obviously are not doing so in a consistent way. Or other hand, the fda could publish the relevant document. Although in each case there would be a requirement for some regulatory changes. The advantages of publication would be weve heard much of this this morning, more informed discussion of everyone of the scientific reasons for the agencys action, which could in turn potentially facilitate drug discovery. There could be in general more transparency on the agencys Decision Making and misperception about the reasons why drugs were not approved and what the results of these pmr studies, any misperceptions could be clarified. And the disadvantages as weve heard from the last discussion theres industry concern that competitors might be advantaged. Theres the potential of release for confidential trade secret information, although i think most of us in this room would agree that the fda is at least as good as it ought to be in protecting that information. Ask finally theres the ever present issue of agency workload. And there, i think, there is a legitimate issue, having once been a person with a workload at fda. You know, i think that what were especially interested in, and i think dr. Gottlieb got at this, is that information that relates to safety and effectiveness much less than information about manufacturing. So i think that information that is most related to drug discovery, most related to clinical outcomes, might be a way to cabin the information such that almost all the information to the public would be released but everything related to the agency could be kept to a reasonable level. Thank you. Can you hear me . Actually, maybe i can turn off the microphone since i no, leave it on . Okay, so i do not have powerpoint. Im so used to speaking with powerpoint im a little intimidated here. Its been a long time. So im using oldfashioned note cards here like back in high school. So my name is eric turner. Sorry, i didnt introduce myself. Im a psychiatrist by training and a former fda reviewer. Im going to speak from a perspective as a clinician and also as a former fda reviewer and consumer of the information as well as serving in production of the information. One theme i want to touch on is not just the qualitative, the types of things that are disclosed but also the manner in which its disclosed and get at the idea of transparency versus tran translucency. Transparency, indeed, is a continuum. Its not an either or thing. One thing id talk about was the drug approval practices. I was told when was an fda reviewer when the works of my labor was complete and the drug were approved then my review would be up on the internet for the world to see. And that was kind of sobering. I knew i was charged with an important job, and if i messed up it would be there for the world to see. So one of my first reviews that i worked on, i think i can say it now because as i learned earlier today some things that become public is no longer confidential information. This is public. I was given an mda application to work on a new drug roboxtan. I started working on it in 98. Gosh, that was a long time ago. And i was immediately struck by the quantity of negative studies. I had never seen this before. I had done a fellowship at National Institutes of health and thought i was at mecca there, and thought i should have access to everything one would need it know about drugs. But i found out i really had been in the dark. But my eyes were opened up there in my fda experience and seeing all these negative studies. And i mentioned to my boss, is this a fluke, and he said no, this happens all the time. In a way i could see the humor in it, but at the same time it was disturbing that in my other hat, that in private practice i should be a clinician and would have been if not for my fda position would have been totally unaware this was going on and blissfully prescribing these drugs believing tat the drugs always be a placebo and here i was learning this was not always the case. So after leaving the fda i did some formal studies of this phenomenon. And my first such study and one thats perhaps best known is looking at antidepressants. It was published n new england journal of medicine in 2008 and took a cohort of 12 antidepressants and looked at all the studies that had been registered with the fda and then tracked those into published literature and asked two questions. First of all, was the study in question published and if it was published, was it published in a way that was consistent with the fda . And it turned out there was quite a discrepancy between these two views. As a clinician you would get the impression looking at Journal Articles that nearly 100 , it was greater than 90 of the articles conveyed a positive conclusion, the ones that were published. On the other hand, if you were looking at fda reviews, you saw it was about 50 50. So quite a striking difference between nearly 100 and 50 . If youre a person doing systematic reviews, it was a shift in the effect size by a third. You calculate a 0. 3 standard deviations using standard data you calculate 0. 4. Then we later did other studies like this using drugs for anxiety and drugs for schizophrenia and got a similar view of things. And then other people have done these studies, making use of fda data. And i guess what id like to segue to now is its been a bit disappointing that not more has been done with this approach, that the fda data is there. Its on the fda website, but its still strikingly underutilized and i think underappreciated. And 97 is when the reviews started going up online. Yet very few people seem to know about it. And obviously not only lay people but even researchers. I think most people in medical research are unaware of this. There was a study done in denmark serving authors of cochran reviews. And its very highly revered as people who do the most highly revered systematic reviews. He found that 97 have not used regulatory data. This is bad. So why is that . I think part of this is part of a lack of appreciation. Perhaps some people simply didnt know about it. Our training is that we are taught that the published literature is the be all and endall of the most authoritative information on medical interventions. So we might be skeptical about there might be something even better elsewhere. And also i think the fact there are obstacles. If they do know about it, then they find, you know, some well, for lack of a better word, user unfriendliness of the reviews. So to talk about drug approval practice, for instance, first of all you have to you need a number of mouse clicks to even find your way to drugs at fda and then to the drug that youre looking for and then sorting out which one has the efficacy and safety reviews to separate that out for the generics which have no efficacy and safety reviews.