The healthcare.gov software problems
[Note: On 2013-10-27 I discovered this New York Times article dated 2013-10-12:
From the Start, Signs of Trouble at Health Portal
By ROBERT PEAR, SHARON LaFRANIERE and IAN AUSTEN.
Thanks to
http://legalinsurrection.com/2013/10/obamacare-website-troubles-began-long-before-launch/
for pointing this out.
But even so, the complaints I raise below still apply, I think, to much of the front page coverage,
in particular the front page articles in the Washington Post on Thursday and Friday,
October 24 and 25.]
2013-10-25:
Consider the following news article, which seems rather typical of the coverage of this problem:
By Sandhya Somashekhar and Amy Goldstein
Washington Post, 2013-10-24
Private contractors in charge of building the federal online health insurance marketplace testified Thursday that the administration went ahead with the Oct. 1 launch of HealthCare.gov despite insufficient testing.
In their first public remarks since the debut of the problem-ridden insurance exchange, executives of the main IT companies told members of the House Energy and Commerce Committee that full tests of the Web site that should have been carried out months in advance, but began just two weeks before its rollout.
“It was not our decision to go live,” said Cheryl Campbell, senior vice president of CGI Federal, which handled most of the project. She said the decision was made by the Centers for Medicare and Medicaid Services, an agency within the Department of Health and Human Services.
CMS officials declined to spell out why the complete tests started so late.
“This system just wasn’t tested enough,” Julie Bataille, director of CMS’s office of communications, acknowledged to reporters. She repeatedly cited what she called “a compressed time frame” without explaining what that meant or the reasons for it.
...
Now, management of software projects funded by the government, for the government,
is something I know something about,
although my knowledge is from the first half of the 1980s,
and was for a computer-savvy part of DoD.
There was a quite standard procedure for managing such projects.
The government would write a RFP, describing in general terms what they wanted.
The contractor (in the experiences I had the contracts were sole-sourced, as the projects involved upgrading and extending capabilities and systems which the company for which I then worked had already provided the government, and no other company had the necessary detailed knowledge of those systems)
would respond with a proposal,
which included a management plan for the development of the system (or extensions to an existing system).
That management plan, and the proposed development to take place,
MUST have at least three parts, describing the activities which would take place once the contract was awarded:
My point, with regard to the healthcare.gov issue,
is that there MUST (right?) have been a test plan written early in the project,
which was approved by the goverment's COTR (Contracting Officer's Technical Representative),
based on inputs from technically qualified people.
If the government did not have the expertise in-house to evaluate test plans,
it was incumbent on senior government management to hire contractors who could evaluate test plans.
In any case, as both the media and the critics in Congress
are making the big issue (as in the article cited above, and evidently the Congressional hearing on which it reports)
the lack of testing.
But the question they should be asking is:
Where is the test plan?
That would state explicitly early in the project the testing that was judged to be required.
This issue does not seem to have made much of a wave on the internet.
On 2013-10-25 at 2000Z I made a Google search on
healthcare.gov "test plan"
with test plan in quotes as shown.
At least on the first page of results,
there was nothing from any MSM or government source that really addressed this issue.
The closest I could find was
http://blog.fmsinc.com/healthcare-gov-is-a-technological-disaster/
from a private company pointing out the problem,
which only mentions "test plan" about 3/4 through a long web page.
Frankly, I am no surprised in the least by these results produced by the Washington "elite"
which values political correctness uber alles.
We have seen numerous examples of highly competent men being forced out of power
simply because what they had said or done was "offensive" to the WASHINGTON BITCHES.
Perhaps my personal experience might be appropriate to mention here,
as an example of the views of people who really should have known better.
Back in the 1990s, I tried to interest some of the Georgetown University math and computer science faculty
in applying certain mathematical ideas (definitions, examples, techniques) to providing a more precise and well-defined way of describing what systems actually do.
To do such would require making both mathematicians more aware of what computer systems are doing
and software engineers more aware of certain mathematical ideas.
I was struck by the response at least one GU professor made, who I think had tried to interest other professors in these ideas.
He said, and this I think an almost literal quote,
"People [which I took to mean the people he had discussed these ideas with]
think there are two solutions to the problems in software and system development you keep alluding to.
One is through better hardware.
The other is to get better people to work the problems."
Well, I was in no position to argue with him.
I am certainly NOT saying that the ideas I had would have solved the healthcare.gov issues.
Those are, as I tried to show above, at base management issues,
failing to properly scope the problem.
But I think the disregard for software issues
pervades much of the Washington decision-making "elite."
2013-10-27:
Further comments on the government and media's handling of this situation:
Here is what I would like to, and expected to, see with regard to the situation:
Making the schedules, initial and updated, available
would allow people to assess whether adequate time was allotted up front
for the various phases of the project.
What the media is giving is a bunch of excuses.
What is needed is to see how well the government planned and managed the project
from the getgo,
whether they understood what they were asking for.
(This problem of starting with a vague desire for a system by top-level people,
vague specs,
which get refined as the work proceeds,
is all too typical of what happened in DoD in the early days.
I remember well a DoD manager, razor-sharp on the needs of his analysts,
who had been assigned to communicate to the system contractor what those needs were,
saying ruefully after the contractor delivered a system implementing his initial cut on those needs,
saying ruefully in his Irish brogue "If we only had known at the beginning what we know now".
There are
some well-known cartoons showing misguided efforts to describe a tree swing
that are apropos!)
I presume it has improved since the 1980s.
But I am not sure that the media understands the questions that should be asked.
But hey, they sure are politically correct, aren't they?
Note further:
provides copious examples of men who said just the opposite,
that the project was in plenty of trouble,
even earlier than July.
By Amy Goldstein and Juliet Eilperin
Washington Post, 2013-11-03
[The headline puts the central issue as
"How political fear was pitted against technical needs".
Well, if the reporting is accurate, that was certainly a part of the problem.
But what stood out to me was a structural, management issue:
So far as I could tell from reading the story,
there was no one office dedicated to implementing this system,
despite the fact that President Obama repeatedly stressed how important it was.
I have opposed the Democrats on policy grounds.
But I had no idea they were so clueless on the difficulty of building systems,
and the steps necessary to build them.
When the Bush 43 administration took us to war in Iraq in 2003,
they, and much of the media who had failed to warn of the hazards and difficulties
of trying to effect political change in the Muslim world by military means,
were accused, rightly in my opinion, of operating in their own little bubble,
making, among other things, facts to fit policy.
Did not the Obama 44 administration make a very similar error
in their development of the computer system necessary
to fulfill the goals President Obama set out for the Obamacare signup process?
Of neither understanding the difficulties of meeting their goals,
nor putting a structure in place to meet those difficulties.]
Politics not a factor, but neither were firm’s ties to failed projects
By Jerry Markon and Alice Crites
Washington Post, 2013-12-22
CGI Federal, the company responsible for building the problem-plagued Web site for the Affordable Care Act, won the job because of what federal officials deemed a “technically superior” proposal, according to government documents and people familiar with the decision.
Not considered in the 2011 selection process was the history of numerous executives at CGI Federal, who had come from another company that had mishandled at least 20 other government information technology projects more than a decade ago. But federal officials were not required to examine that long-term track record, which included a highly publicized failure to automate retirement benefits for millions of federal workers.
By 2011, CGI Federal already had been cleared to do government work at the Centers for Medicare and Medicaid Services (CMS), the agency overseeing the rollout of the new health-care law.
The company had been included in a pool of pre-screened, approved contractors in 2007, during the George W. Bush administration, and only firms in that pool were later allowed to bid for the Affordable Care Act work. It was at that earlier time that the problems at American Management Systems, the Fairfax County IT contractor acquired by CGI, would have figured into the assessment of CGI Federal, contracting experts say.
In hindsight, one former CMS official said, the AMS record “could well have knocked [CGI Federal] out of the competition, and probably should have.”
In light of the bungled launch of the health-care Web site, critics have repeatedly asked how CGI Federal got the contract, and there has been widespread speculation on online news sites and blogs that the firm was selected because of political ties to the Obama administration.
But a review of internal documents and interviews with former and current federal officials show that the selection process was walled off from politics.
Career officials at CMS,
working out of an obscure Baltimore-based contracting office,
focused in 2011 on the proposals submitted by four bidders,
examining technical competency and cost
while giving relatively little attention to past performance.
Since 2007,
CGI Federal had won 17 other CMS jobs under the pre-screened contract,
worth several hundred million dollars.
The review of CGI Federal concluded there was “no evidence of performance risk,” according to a CMS document, portions of which were read to The Washington Post by a person familiar with the contracting process.
CGI Federal won the $93 million contract, now worth $293 million, in September 2011.
...
In the summer of 2011, the Obama administration was racing to figure out who would build the Affordable Care Act’s all-important Web site. “Time is of the essence,” said an internal CMS draft request for proposals, dated June 24, 2011.
With deadlines looming, officials limited the bidders to the winners of the 2007 umbrella contract. “There was very great time pressure,” recalled a former CMS official familiar with the process. “The feeling was we had a lot of big IT powerhouses and they knew our work.’’
Four firms applied: IBM, Computer Sciences Corp., Quality Software Services and CGI Federal.
As is tradition, the selection process was strictly sheltered from political considerations and political appointees at CMS or elsewhere in the administration, according to numerous former and current CMS employees.
“I deliberately stayed out of contracting,’’ said Leslie Norwalk, the agency’s acting administrator under President George W. Bush. “If that had changed, I would have heard rumblings.’’
Agency documents show that a career contracting officer, in the CMS Office of Acquisition and Grants Management, was in charge of awarding the contract for the Web site. The selection process focused on bidders’ business and technical proposals, which described the proposed cost, hardware and software required, proposed infrastructure design, security requirements, and key personnel.
The companies’ past performance was not a priority. The CMS document listed it as the second least important of seven technical evaluation factors. And the review would look only at past work under the 2007 umbrella contract, the document said.
While the review found no performance risk for CGI Federal, agency officials had not always been satisfied with the company’s work. In 2010, CMS rejected CGI Federal’s bid under the umbrella contract to perform work on four health IT systems, in part citing unspecified performance issues in carrying out an earlier contract, according to a Government Accountability Office ruling. CGI Federal protested the CMS decision, but the GAO upheld it.
And the review did not take into account problems that several other CGI projects had encountered, including years of delay and system failures on a state computer system in Hawaii. State auditors in 2010 partially faulted another CGI subsidiary, CGI Technologies and Solutions, for those issues.
CGI says its overall work is high-quality and nearly always on time and on budget, and the company has also had many successes. In the past two years, CGI Federal has been awarded contracts with at least 25 federal agencies.
Documents and interviews show that CGI Federal was selected because of what was deemed to be the company’s technical prowess, even though the price of the firm’s proposal was higher than that of at least of one competitor.
By Amy Goldstein February 23 at 12:01 AM
Washington Post, 2016-02-23
During the two years before the disastrous opening of HealthCare.gov, federal officials in charge of creating the online insurance marketplace received 18 written warnings that the mammoth project was mismanaged and off course but never considered postponing its launch, according to government investigators.
The warnings included a series of 11 scathing reviews from an outside consultant — among them a top-10 list of risks drawn up in the spring of 2013 that cited inadequate planning for the website’s capacity and deviations from usual IT standards. A few months before, then-Health and Human Services Secretary Kathleen Sebelius had hired another consultant to review the project and recommend ways to improve its management, but its advice was never shared with the technical staff working on the website.
The long trail of unheeded warnings is among the findings from an exhaustive two-year inquiry by HHS’s Office of Inspector General into the failings of HealthCare.gov, which crashed within two hours of its launch on Oct. 1, 2013. The failings tarnished the start of a central aspect of the Affordable Care Act — new insurance marketplaces for Americans who cannot get affordable coverage through a job — and embarrassed the White House, which championed the law.
The findings are contained in a “case study” to be released Tuesday. It represents the most penetrating look ever into what went wrong with the building of the federal insurance exchange and what was done to fix it. It is based on interviews with 86 employees of HHS, its Centers for Medicare and Medicaid Services (CMS) and companies that worked on the project, as well as on several thousand emails, memos, government contracts and other internal documents.
Many of the basic contours laid out in the 84-page report are, by now, familiar: Federal health officials failed to recognize the enormity of the undertaking, were disorganized and fragmented, were hampered by late and shifting ACA policies, had too little money, used poor contracting practices, and ignored problems until it was too late.
But the inquiry unearthed vivid details that have not been public. And it concludes that the central reason for the problems rested not with the shoddy work of vital IT contractors but with mismanagement by federal health officials carrying out this part of the law.
“CMS didn’t need a technical surge, they needed an organizational surge,” an agency employee told the investigators. A lack of leadership, the report says, “caused delays in decision-\making, lack of clarity in project tasks and the inability of CMS to recognize the magnitude of problems as the project deteriorated.”
The investigators also concluded that once the crisis erupted (only six people nationwide managed to select health plans through HealthCare.gov on its first day) the initial repair blitz was not primarily the result of the “tech surge” ballyhooed by the White House — the talent quickly imported from Silicon Valley and other leading IT firms. The turnaround was fostered mainly by an abrupt culture shift in which government workers, contractors and the tech imports worked hand in hand, initially at a command center in Herndon, Va, Va. Within two months, about 4 in 5 consumers could use the website.
Over the past two years, top federal health officials have apologized for HealthCare.gov’s troubled start, and President Obama has called it a “well-documented disaster.” CMS spokesman Aaron Albright said the problems cited in the report taught the agency lessons about “leadership, accountability and prioritization” that it has applied to its work since.
Among the report’s revelations:
●In the summer of 2013, CMS officials asked CGI Federal, the main contractor building HealthCare.gov, to demonstrate a simple feature called Account Lite, intended to let consumers create accounts before enrollment began. CGI was behind schedule and, when it finally did the demonstration, federal workers found 105 defects.
●On Sept. 26, five days before the launch, CMS officials discovered that the website had capacity for just a fraction of the planned number of consumers who could shop for health plans and fill out applications. That afternoon, CMS officials drove to the Laurel, Md., offices of a contractor, Terremark, and ordered its managers to double the capacity within 72 hours.
●Another contractor, QSSI, which was building the system for consumers to create accounts and verify their identities, underestimated the capacity required because its leaders did not know that consumers would not be able to browse health plans unless they first created an account. Just after midnight on Oct. 1, when HealthCare.gov began to run, QSSI’s staff members were in their office, watching as “everything was turning red on our screens,” indicating that people couldn’t get onto the site, the report says.
Before the site opened, CMS had not tested it end to end to see how the parts worked together. “You can’t test what is not built,” a contractor told the investigators.
Such last-minute chaos stemmed from decisions and dynamics that had begun much earlier, the investigators said. The ACA provided HHS with $1 billion for the administrative expenses of implementing the law, but the department “ceded over half these funds” to the Internal Revenue Service and other agencies carrying out parts of the law, aggravating its own financial strain.
And delays occurred, in part, because of close scrutiny of the CMS’s work by the White House staff, even on relatively minor issues. CMS employees were frustrated, the report says, “with the discussion around changing the term ‘nationwide health insurance’ to “health insurance’ in official documents.”
Work was hampered, too, by turnover of key staff at the CMS Center for Consumer Information and Insurance Oversight, which oversees the marketplace.
As the launch date neared, the report says, CMS officials and workers became “desensitized to bad news about progress,” doing little to respond to warnings and remaining too optimistic.
By the next spring, that had changed. On April 1, 2014, the day after the first enrollment season ended, the agency’s leaders met for three days of “ruthless prioritization” for the second sign-up period that fall. They listed the unfinished work on the website and, after considering what each item would require, cut the list in half.
Even today, after 9.6 million people nationwide signed up during the third enrollment for ACA coverage, HealthCare.gov “faces ongoing challenges,” the report says, including the completion of the website. One last part is an automated payment system with insurers. CMS began to use the system last month, starting with insurers whose own IT systems were ready for it.
From the Start, Signs of Trouble at Health Portal
By ROBERT PEAR, SHARON LaFRANIERE and IAN AUSTEN.
Thanks to
http://legalinsurrection.com/2013/10/obamacare-website-troubles-began-long-before-launch/
for pointing this out.
But even so, the complaints I raise below still apply, I think, to much of the front page coverage,
in particular the front page articles in the Washington Post on Thursday and Friday,
October 24 and 25.]
2013-10-25:
Consider the following news article, which seems rather typical of the coverage of this problem:
2013-10-24-WP-Somashekhar-Goldstein-house-panel-grills-contractors-on-troubled-health-insurance-web-site
Full testing of HealthCare.gov began too late, contractors sayBy Sandhya Somashekhar and Amy Goldstein
Washington Post, 2013-10-24
Private contractors in charge of building the federal online health insurance marketplace testified Thursday that the administration went ahead with the Oct. 1 launch of HealthCare.gov despite insufficient testing.
In their first public remarks since the debut of the problem-ridden insurance exchange, executives of the main IT companies told members of the House Energy and Commerce Committee that full tests of the Web site that should have been carried out months in advance, but began just two weeks before its rollout.
“It was not our decision to go live,” said Cheryl Campbell, senior vice president of CGI Federal, which handled most of the project. She said the decision was made by the Centers for Medicare and Medicaid Services, an agency within the Department of Health and Human Services.
CMS officials declined to spell out why the complete tests started so late.
“This system just wasn’t tested enough,” Julie Bataille, director of CMS’s office of communications, acknowledged to reporters. She repeatedly cited what she called “a compressed time frame” without explaining what that meant or the reasons for it.
...
Now, management of software projects funded by the government, for the government,
is something I know something about,
although my knowledge is from the first half of the 1980s,
and was for a computer-savvy part of DoD.
There was a quite standard procedure for managing such projects.
The government would write a RFP, describing in general terms what they wanted.
The contractor (in the experiences I had the contracts were sole-sourced, as the projects involved upgrading and extending capabilities and systems which the company for which I then worked had already provided the government, and no other company had the necessary detailed knowledge of those systems)
would respond with a proposal,
which included a management plan for the development of the system (or extensions to an existing system).
That management plan, and the proposed development to take place,
MUST have at least three parts, describing the activities which would take place once the contract was awarded:
- Firming up the requirements and detailed design of the system,
resulting in requirements and design documents.
These documents would be submitted to the gov for their review and approval. - The actual system build (the implementation phase).
- The testing that would be required, described in a TEST PLAN.
This plan also would be submitted to the gov early in the life of the contract,
well in advance of the actual testing to take place,
for their (the government's) approval.
My point, with regard to the healthcare.gov issue,
is that there MUST (right?) have been a test plan written early in the project,
which was approved by the goverment's COTR (Contracting Officer's Technical Representative),
based on inputs from technically qualified people.
If the government did not have the expertise in-house to evaluate test plans,
it was incumbent on senior government management to hire contractors who could evaluate test plans.
In any case, as both the media and the critics in Congress
are making the big issue (as in the article cited above, and evidently the Congressional hearing on which it reports)
the lack of testing.
But the question they should be asking is:
Where is the test plan?
That would state explicitly early in the project the testing that was judged to be required.
This issue does not seem to have made much of a wave on the internet.
On 2013-10-25 at 2000Z I made a Google search on
healthcare.gov "test plan"
with test plan in quotes as shown.
At least on the first page of results,
there was nothing from any MSM or government source that really addressed this issue.
The closest I could find was
http://blog.fmsinc.com/healthcare-gov-is-a-technological-disaster/
from a private company pointing out the problem,
which only mentions "test plan" about 3/4 through a long web page.
Frankly, I am no surprised in the least by these results produced by the Washington "elite"
which values political correctness uber alles.
We have seen numerous examples of highly competent men being forced out of power
simply because what they had said or done was "offensive" to the WASHINGTON BITCHES.
Perhaps my personal experience might be appropriate to mention here,
as an example of the views of people who really should have known better.
Back in the 1990s, I tried to interest some of the Georgetown University math and computer science faculty
in applying certain mathematical ideas (definitions, examples, techniques) to providing a more precise and well-defined way of describing what systems actually do.
To do such would require making both mathematicians more aware of what computer systems are doing
and software engineers more aware of certain mathematical ideas.
I was struck by the response at least one GU professor made, who I think had tried to interest other professors in these ideas.
He said, and this I think an almost literal quote,
"People [which I took to mean the people he had discussed these ideas with]
think there are two solutions to the problems in software and system development you keep alluding to.
One is through better hardware.
The other is to get better people to work the problems."
Well, I was in no position to argue with him.
I am certainly NOT saying that the ideas I had would have solved the healthcare.gov issues.
Those are, as I tried to show above, at base management issues,
failing to properly scope the problem.
But I think the disregard for software issues
pervades much of the Washington decision-making "elite."
2013-10-27:
Further comments on the government and media's handling of this situation:
Here is what I would like to, and expected to, see with regard to the situation:
- Who was managing the project?
The news reports I have seen say it was handled by the Centers for Medicare & Medicaid Services.
But where within that?
That's obviously, at the top, primarily a bunch of medical experts and economist,
not computer types.
Who (what office, what manager) within that was running the project? - Any project has a schedule.
There is an initial schedule at the onset of the project,
then as the project proceeds almost inevitably there will be update to that schedule.
Where can the public see both the initial schedule and the event-caused updates?
There can be no excuse for not making that information publicly available.
If it were a DoD project or intel project, there would be security and classification issues.
But this project surely isn't classified.
Making the schedules, initial and updated, available
would allow people to assess whether adequate time was allotted up front
for the various phases of the project.
What the media is giving is a bunch of excuses.
What is needed is to see how well the government planned and managed the project
from the getgo,
whether they understood what they were asking for.
(This problem of starting with a vague desire for a system by top-level people,
vague specs,
which get refined as the work proceeds,
is all too typical of what happened in DoD in the early days.
I remember well a DoD manager, razor-sharp on the needs of his analysts,
who had been assigned to communicate to the system contractor what those needs were,
saying ruefully after the contractor delivered a system implementing his initial cut on those needs,
saying ruefully in his Irish brogue "If we only had known at the beginning what we know now".
There are
some well-known cartoons showing misguided efforts to describe a tree swing
that are apropos!)
I presume it has improved since the 1980s.
But I am not sure that the media understands the questions that should be asked.
But hey, they sure are politically correct, aren't they?
Note further:
Marilyn B. Tavenner, the administrator of the Centers for Medicare and Medicaid Services,The rest of the NYT article linked to above
and Kathleen Sebelius, the secretary of health and human services,
both insisted in July that the project was not in trouble.
provides copious examples of men who said just the opposite,
that the project was in plenty of trouble,
even earlier than July.
2013-11-03-WP-TeamPC-builds-a-system
HealthCare.gov: How political fear was pitted against technical needsBy Amy Goldstein and Juliet Eilperin
Washington Post, 2013-11-03
[The headline puts the central issue as
"How political fear was pitted against technical needs".
Well, if the reporting is accurate, that was certainly a part of the problem.
But what stood out to me was a structural, management issue:
So far as I could tell from reading the story,
there was no one office dedicated to implementing this system,
despite the fact that President Obama repeatedly stressed how important it was.
I have opposed the Democrats on policy grounds.
But I had no idea they were so clueless on the difficulty of building systems,
and the steps necessary to build them.
When the Bush 43 administration took us to war in Iraq in 2003,
they, and much of the media who had failed to warn of the hazards and difficulties
of trying to effect political change in the Muslim world by military means,
were accused, rightly in my opinion, of operating in their own little bubble,
making, among other things, facts to fit policy.
Did not the Obama 44 administration make a very similar error
in their development of the computer system necessary
to fulfill the goals President Obama set out for the Obamacare signup process?
Of neither understanding the difficulties of meeting their goals,
nor putting a structure in place to meet those difficulties.]
2013-12-22-WP-healthcaregov-contract-politics-not-a-factor-but-neither-were-firms-ties-to-failed-projects
HealthCare.gov contract:Politics not a factor, but neither were firm’s ties to failed projects
By Jerry Markon and Alice Crites
Washington Post, 2013-12-22
CGI Federal, the company responsible for building the problem-plagued Web site for the Affordable Care Act, won the job because of what federal officials deemed a “technically superior” proposal, according to government documents and people familiar with the decision.
Not considered in the 2011 selection process was the history of numerous executives at CGI Federal, who had come from another company that had mishandled at least 20 other government information technology projects more than a decade ago. But federal officials were not required to examine that long-term track record, which included a highly publicized failure to automate retirement benefits for millions of federal workers.
By 2011, CGI Federal already had been cleared to do government work at the Centers for Medicare and Medicaid Services (CMS), the agency overseeing the rollout of the new health-care law.
The company had been included in a pool of pre-screened, approved contractors in 2007, during the George W. Bush administration, and only firms in that pool were later allowed to bid for the Affordable Care Act work. It was at that earlier time that the problems at American Management Systems, the Fairfax County IT contractor acquired by CGI, would have figured into the assessment of CGI Federal, contracting experts say.
In hindsight, one former CMS official said, the AMS record “could well have knocked [CGI Federal] out of the competition, and probably should have.”
In light of the bungled launch of the health-care Web site, critics have repeatedly asked how CGI Federal got the contract, and there has been widespread speculation on online news sites and blogs that the firm was selected because of political ties to the Obama administration.
But a review of internal documents and interviews with former and current federal officials show that the selection process was walled off from politics.
Career officials at CMS,
working out of an obscure Baltimore-based contracting office,
focused in 2011 on the proposals submitted by four bidders,
examining technical competency and cost
while giving relatively little attention to past performance.
Since 2007,
CGI Federal had won 17 other CMS jobs under the pre-screened contract,
worth several hundred million dollars.
The review of CGI Federal concluded there was “no evidence of performance risk,” according to a CMS document, portions of which were read to The Washington Post by a person familiar with the contracting process.
CGI Federal won the $93 million contract, now worth $293 million, in September 2011.
...
In the summer of 2011, the Obama administration was racing to figure out who would build the Affordable Care Act’s all-important Web site. “Time is of the essence,” said an internal CMS draft request for proposals, dated June 24, 2011.
With deadlines looming, officials limited the bidders to the winners of the 2007 umbrella contract. “There was very great time pressure,” recalled a former CMS official familiar with the process. “The feeling was we had a lot of big IT powerhouses and they knew our work.’’
Four firms applied: IBM, Computer Sciences Corp., Quality Software Services and CGI Federal.
As is tradition, the selection process was strictly sheltered from political considerations and political appointees at CMS or elsewhere in the administration, according to numerous former and current CMS employees.
“I deliberately stayed out of contracting,’’ said Leslie Norwalk, the agency’s acting administrator under President George W. Bush. “If that had changed, I would have heard rumblings.’’
Agency documents show that a career contracting officer, in the CMS Office of Acquisition and Grants Management, was in charge of awarding the contract for the Web site. The selection process focused on bidders’ business and technical proposals, which described the proposed cost, hardware and software required, proposed infrastructure design, security requirements, and key personnel.
The companies’ past performance was not a priority. The CMS document listed it as the second least important of seven technical evaluation factors. And the review would look only at past work under the 2007 umbrella contract, the document said.
While the review found no performance risk for CGI Federal, agency officials had not always been satisfied with the company’s work. In 2010, CMS rejected CGI Federal’s bid under the umbrella contract to perform work on four health IT systems, in part citing unspecified performance issues in carrying out an earlier contract, according to a Government Accountability Office ruling. CGI Federal protested the CMS decision, but the GAO upheld it.
And the review did not take into account problems that several other CGI projects had encountered, including years of delay and system failures on a state computer system in Hawaii. State auditors in 2010 partially faulted another CGI subsidiary, CGI Technologies and Solutions, for those issues.
CGI says its overall work is high-quality and nearly always on time and on budget, and the company has also had many successes. In the past two years, CGI Federal has been awarded contracts with at least 25 federal agencies.
Documents and interviews show that CGI Federal was selected because of what was deemed to be the company’s technical prowess, even though the price of the firm’s proposal was higher than that of at least of one competitor.
2016
2016-02-23-WP-hhs-failed-to-heed-many-warnings-that-healthcaregov-was-in-trouble
HHS failed to heed many warnings that HealthCare.gov was in troubleBy Amy Goldstein February 23 at 12:01 AM
Washington Post, 2016-02-23
During the two years before the disastrous opening of HealthCare.gov, federal officials in charge of creating the online insurance marketplace received 18 written warnings that the mammoth project was mismanaged and off course but never considered postponing its launch, according to government investigators.
The warnings included a series of 11 scathing reviews from an outside consultant — among them a top-10 list of risks drawn up in the spring of 2013 that cited inadequate planning for the website’s capacity and deviations from usual IT standards. A few months before, then-Health and Human Services Secretary Kathleen Sebelius had hired another consultant to review the project and recommend ways to improve its management, but its advice was never shared with the technical staff working on the website.
The long trail of unheeded warnings is among the findings from an exhaustive two-year inquiry by HHS’s Office of Inspector General into the failings of HealthCare.gov, which crashed within two hours of its launch on Oct. 1, 2013. The failings tarnished the start of a central aspect of the Affordable Care Act — new insurance marketplaces for Americans who cannot get affordable coverage through a job — and embarrassed the White House, which championed the law.
The findings are contained in a “case study” to be released Tuesday. It represents the most penetrating look ever into what went wrong with the building of the federal insurance exchange and what was done to fix it. It is based on interviews with 86 employees of HHS, its Centers for Medicare and Medicaid Services (CMS) and companies that worked on the project, as well as on several thousand emails, memos, government contracts and other internal documents.
Many of the basic contours laid out in the 84-page report are, by now, familiar: Federal health officials failed to recognize the enormity of the undertaking, were disorganized and fragmented, were hampered by late and shifting ACA policies, had too little money, used poor contracting practices, and ignored problems until it was too late.
But the inquiry unearthed vivid details that have not been public. And it concludes that the central reason for the problems rested not with the shoddy work of vital IT contractors but with mismanagement by federal health officials carrying out this part of the law.
“CMS didn’t need a technical surge, they needed an organizational surge,” an agency employee told the investigators. A lack of leadership, the report says, “caused delays in decision-\making, lack of clarity in project tasks and the inability of CMS to recognize the magnitude of problems as the project deteriorated.”
The investigators also concluded that once the crisis erupted (only six people nationwide managed to select health plans through HealthCare.gov on its first day) the initial repair blitz was not primarily the result of the “tech surge” ballyhooed by the White House — the talent quickly imported from Silicon Valley and other leading IT firms. The turnaround was fostered mainly by an abrupt culture shift in which government workers, contractors and the tech imports worked hand in hand, initially at a command center in Herndon, Va, Va. Within two months, about 4 in 5 consumers could use the website.
Over the past two years, top federal health officials have apologized for HealthCare.gov’s troubled start, and President Obama has called it a “well-documented disaster.” CMS spokesman Aaron Albright said the problems cited in the report taught the agency lessons about “leadership, accountability and prioritization” that it has applied to its work since.
Among the report’s revelations:
●In the summer of 2013, CMS officials asked CGI Federal, the main contractor building HealthCare.gov, to demonstrate a simple feature called Account Lite, intended to let consumers create accounts before enrollment began. CGI was behind schedule and, when it finally did the demonstration, federal workers found 105 defects.
●On Sept. 26, five days before the launch, CMS officials discovered that the website had capacity for just a fraction of the planned number of consumers who could shop for health plans and fill out applications. That afternoon, CMS officials drove to the Laurel, Md., offices of a contractor, Terremark, and ordered its managers to double the capacity within 72 hours.
●Another contractor, QSSI, which was building the system for consumers to create accounts and verify their identities, underestimated the capacity required because its leaders did not know that consumers would not be able to browse health plans unless they first created an account. Just after midnight on Oct. 1, when HealthCare.gov began to run, QSSI’s staff members were in their office, watching as “everything was turning red on our screens,” indicating that people couldn’t get onto the site, the report says.
Before the site opened, CMS had not tested it end to end to see how the parts worked together. “You can’t test what is not built,” a contractor told the investigators.
Such last-minute chaos stemmed from decisions and dynamics that had begun much earlier, the investigators said. The ACA provided HHS with $1 billion for the administrative expenses of implementing the law, but the department “ceded over half these funds” to the Internal Revenue Service and other agencies carrying out parts of the law, aggravating its own financial strain.
And delays occurred, in part, because of close scrutiny of the CMS’s work by the White House staff, even on relatively minor issues. CMS employees were frustrated, the report says, “with the discussion around changing the term ‘nationwide health insurance’ to “health insurance’ in official documents.”
Work was hampered, too, by turnover of key staff at the CMS Center for Consumer Information and Insurance Oversight, which oversees the marketplace.
As the launch date neared, the report says, CMS officials and workers became “desensitized to bad news about progress,” doing little to respond to warnings and remaining too optimistic.
By the next spring, that had changed. On April 1, 2014, the day after the first enrollment season ended, the agency’s leaders met for three days of “ruthless prioritization” for the second sign-up period that fall. They listed the unfinished work on the website and, after considering what each item would require, cut the list in half.
Even today, after 9.6 million people nationwide signed up during the third enrollment for ACA coverage, HealthCare.gov “faces ongoing challenges,” the report says, including the completion of the website. One last part is an automated payment system with insurers. CMS began to use the system last month, starting with insurers whose own IT systems were ready for it.
posted by KHarbaugh at 12:00
<< Home